Telecommunication and multimedia management method and apparatus

ABSTRACT

A telecommunication and multimedia management apparatus and method that supports voice and other media communications and that enables users to: (i) participate in multiple conversation modes, including live phone calls, conference calls, instant voice messaging or tactical communications; (ii) review the messages of conversations in either a live mode or a time-shifted mode and to seamlessly transition back and forth between the two modes; (iii) participate in multiple conversations either concurrently or simultaneously; (iv) archive the messages of conversations for later review or processing; and (v) persistently store media either created or received on the communication devices of users. The latter feature enables users to generate or review media when either disconnected from the network or network conditions are poor and to optimize the delivery of media over the network based on network conditions and the intention of the users participating in conversations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.17/062,060 filed Oct. 2, 2020, which is a continuation of U.S.application Ser. No. 16/669,865 filed on Oct. 31, 2019 (now U.S. Pat.No. 10,841,261), which is a Continuation of U.S. application Ser. No.16/161,474 filed on Oct. 16, 2018 (now U.S. Pat. No. 10,511,557), whichis a Continuation of Ser. No. 15/937,361, filed on Mar. 27, 2018 (nowU.S. Pat. No. 10,158,591), which is a Continuation of Ser. No.15/923,287, filed on Mar. 16, 2018 (now U.S. Pat. No. 10,129,191), whichis a Continuation of Ser. No. 15/584,224, filed on May 2, 2017 (now U.S.Pat. No. 10,142,270). U.S. application Ser. No. 15/584,224 is aContinuation of U.S. application Ser. No. 15/231,240, filed on Aug. 8,2016 (now U.S. Pat. No. 9,674,122), which is a Continuation of U.S.application Ser. No. 14/799,528, filed Jul. 14, 2015 (now U.S. Pat. No.9,456,087). U.S. application Ser. No. 14/799,528 is a Continuation ofU.S. application Ser. No. 14/198,043, filed on Mar. 5, 2014 (now U.S.Pat. No. 9,154,628), which is a Continuation of U.S. application Ser.No. 13/555,034 (now U.S. Pat. No. 8,705,714), filed Jul. 20, 2012, whichis a Continuation of U.S. application Ser. No. 12/037,256 (now U.S. Pat.No. 8,243,894), filed on Feb. 26, 2008, which is Continuation of Ser.No. 12/028,400 (now U.S. Pat. No. 8,180,029), filed on Feb. 8, 2008,which claims the benefit of U.S. Provisional Patent Application Nos.60/937,552, filed Jun. 28, 2007 and 60/999,619, filed on Oct. 19, 2007.A1l of the foregoing applications are incorporated herein by referencein their entirety for all purposes.

BACKGROUND Field of the Invention

This invention pertains to telecommunications, and more particularly, toa telecommunication and multimedia management method and apparatus thatenables users to review the messages of conversations in either a livemode or a time-shifted mode and to transition the conversation back andforth between the two modes, participate in multiple conversations andto archive the messages of conversations for later review or processing.

Description of Related Art

The current state of voice communications suffers from inertia. In spiteof automated switching, high bandwidth networks and technologies such assatellites, fiber optics, Voice over IP (VoIP), wireless and cellularnetworks, there has been little change in how people use telephones. Oneis still required to pick up the phone, dial another party, wait for aconnection to be made, and then engage in a full-duplex, synchronousconversation with the dialed party. If the recipient does not answer, noconnection is made, and the conversation does not take place.

At best, a one-way asynchronous voice message may be left if therecipient has voice mail. The process of delivering the voice mail,however, is burdensome and time consuming The caller is required to waitfor the phone on the other end to stop ringing, transition into thevoice mail system, listen to a voice message greeting, and then leavethe message. Current voice mail systems are also inconvenient for therecipient. The recipient has to dial a code to access their voice mail,navigate through a series of prompts, listen to any earlier receivedvoice messages in the queue, and then finally listen to the message ofthe sender.

Another drawback with typical voice mail systems is the inability toorganize or permanently archive voice messages. With some voice mailsystems, a user may save a message, but it is automatically deletedafter a predetermined period of time and lost forever.

Yet another problem with current voice mail systems is that a connectionmust be made between the caller and the voice mail system before amessage can be left. If no connection is made, there is no way for thecaller to leave a message.

Current telephone systems are based on relatively simplistic usagepatterns: real-time live calls or disjointed voice mail messages, whichare typically deleted as they are heard. These forms of voicecommunications do not capture the real power that can be achieved withvoice communication or take advantage of the advances of network speedand bandwidth that is now available. A1so, if the phone network is down,or is inaccessible, (e.g., a cell phone user is in an area of nocoverage or the phone lines are down due to bad weather), nocommunication can take place.

In general, telephone based communications have not kept pace with theadvances in text-based communications. Instant messaging, emailing,faxing, chat groups, and the ability to archive text messages, are allcommonplace with text based communications. Other than voice mail, thereare few existing tools available to manage and/or archive voicemessages. In comparison, the tools currently available to managetelephone communications are primitive compared to text communications.

The corporate environment provides just one example of the weakness incurrent voice communication tools. There is currently no integrated wayto manage voice communications across an organization as a corporateasset. Employees generally do not record or persistently store theirphone conversations. Most business related voice communication assetsare gone as quickly as the words are spoken, with no way to manage orstore the content of those conversations in any manageable form.

As an illustrative example, consider a sales executive at a company.During the course of a busy day, the executive may make a number ofcalls, and close several sales, with customers over the phone. Withoutthe ability to organize, store, and later retrieve these conversations,there is no way for the executive to resolve potential issues that mayarise, such as recalling the terms of one deal versus another, orchallenging a customer who disputes the terms of a previously agreedupon sale. If this executive had the ability to easily retrieve andreview conversations, these types of issues could be easily andfavorably resolved.

Current tactical radio systems, such as those used by the military,fire, police, paramedics, rescue teams, and first responders, alsosuffer from a number of deficiencies. Most tactical radio communicationmust occur through a “live” radio connection between the sender of amessage and a recipient. If there is no radio connection between the twoparties, there can be no communication. Urgent messages cannot be sentif either the sender or the receiver does not have access to theirradio, or a radio circuit connection is not established. Tacticalcommunications are therefore plagued with several basic problems. Thereis no way (i) to guarantee the delivery of messages, (ii) for arecipient to go back and listen to a message that was not heard in realtime; (iii) to control the granularity of the participants in aconversation; (iv) for the system to cope when there is a lack of signalintegrity for a live conversation. If a message is not heard live, it ismissed. There are no tools for either the sender or a recipient tomanage, prioritize, archive and later retrieve (i.e. time-shift) themessages of a conversation that were previously sent.

Yet another drawback with tactical radio communication systems is thatonly one message can be sent at a time per channel. Consider an exampleof a large building fire, where multiple teams of fire fighters, policeand paramedics are simultaneously rescuing victims trapped in thebuilding, fighting the fire, providing medical aid to victims, andcontrolling bystanders. If each of the teams is using the same channel,communications may become crowded and chaotic. Transmissions get“stepped on” when more than one person is transmitting at the same time.A1so there is no way to differentiate between high and low prioritymessages. A team inside the burning building fighting the fire orrescuing trapped victims should have a higher priority over other teams,such as those controlling bystanders. If high priority messages arestepped on by lower priority messages, it could not only hamperimportant communications, but could endanger the lives of the firefighters and victims in the building.

One possible solution to the lack of ability to prioritize messages isto use multiple channels, where each team is assigned a differentchannel. This solution, however, creates its own set of problems. Howdoes the fire chief determine which channel to listen too at any pointin time? How do multiple teams communicate with one another if they areall on different channels? If one team calls for urgent help, how areother teams to know if they are listening to other channels? Whilemultiple channels can alleviate some issues, it can also causeconfusion, creating more problems than if a single channel is used.

The lack of management tools that effectively prioritize messages, thatallow multiple conversations to take place at the same time, that enablethe time-shifting of messages to guarantee delivery, or that supportarchiving and storing conversations for later retrieval and review, allcontribute to the problems associated with tactical radios. In firstresponder situations, such as with the military, police, and fire,effective communication tools can literally mean the difference betweenlife and death, or the success or failure of a mission. The aboveburning building example is useful in illustrating just some of theissues with current tactical radio communications. Similar problemsexist with the military, police, first responders and others who usetactical communications.

With packet-based networks, commonly used protocols include TransmissionControl Protocol (TCP) and User Datagram Protocol (UDP). UDP offers theadvantage of fast delivery of data, but at the expense of completeness.Packets may be dropped in transit and not available when attempting torender the data as soon as possible at the destination. In spite of theshortcomings, UDP is the standard for Voice over Internet Protocol(VoIP) transmissions due to its speed attributes. On the other hand TCPdoes guarantee the delivery of perfect (i.e., an exact copy of thetransmitted data) data, but at the expense of latency. A1l packets aredelivered, regardless of how long it takes. This delay makes TCPimpractical for use with “live” phone calls. Currently there are noknown protocols that offer the performance advantages of both TCP andUDP, where “good enough” media can be transmitted for rendering as soonas possible, with the eventual delivery of a perfect copy of the media.A1so there is no protocol that determines how much information should besent over the network based on the presence of recipients on the networkand their intentions to render the data either live or in a time-shiftedmode. In addition, other factors commonly considered, such as networklatency, network degradation, packet loss, packet damage, and generalbandwidth conditions, are used in determining how much data to transmit.Prior art systems, however, do not consider the presence and intentionsof recipients. As a result, the default assumption is that the data isrendered by the recipient in real time. When a recipient is not going torender data immediately, these prior art systems unnecessarily usebandwidth when it is not needed, degrading the overall performance ofthe network.

For the reasons recited above, telephone, voicemail and tactical voicecommunications systems are inadequate. An improved voice and mediacommunication and management system and method, and improvements indelivering voice and other media over packet-based networks, istherefore needed.

SUMMARY OF THE INVENTION

The above-described problems are solved by a communication application,method and apparatus that supports new modes of engaging inconversations and/or managing multiple conversations using a variety ofmedia types, such as voice, video, text, location, sensor information,and other data. Users can engage in conversations by sending messages todesignated recipients. Depending on preferences and priorities, therecipient(s) might participate in the conversation in real time, or theymight simply be notified that the message is ready for retrieval. In thelatter case, the recipient participates in the conversation in atime-shifted mode by reviewing and replying to the recorded message attheir convenience.

Users are empowered to conduct communications in either: (i) anear-synchronous or “live” conversation, providing a user experiencesimilar to a standard full duplex phone call; or (ii) in a series ofback and forth time-delayed transmissions (i.e., time-shifted mode).Further, users engaged in a conversation can seamlessly transition fromthe live mode to the time-shifted mode and back again. This attributealso makes it possible for users to engage in multiple conversations, atthe same time, by prioritizing and shifting between the two modes foreach conversation. Two individuals using the system can therefore sendrecorded voice messages back and forth to each other and review themessages when convenient, or the messages can be sent at a rate wherethey essentially merge into a live, synchronous voice conversation. Thisnew form of communication, for the purposes of the present application,is referred to as “Voxing.”

When you “Vox” someone, the conversation consists of a series ofdiscrete recorded messages, which are recorded in a number of locations,which may include the encoding device of the sender, (e.g. a phone orcomputer), servers on multiple transmission hops across the network, andthe receiver's rendering device. Unlike a standard phone call or voicemail, the system provides the following features and advantages: (i) theconversation can transition between live and time-shifted or vice versa;(ii) the discrete messages of the conversation are semantically threadedtogether and archived; (iii) since the messages are recorded and areavailable for later retrieval, attention can be temporarily divertedfrom the conversation and then the conversation can be later reviewedwhen convenient; (iv) the conversation can be paused for seconds,minutes, hours, or even days, and can be picked up again where left off;(v) one can rejoin a conversation in progress and rapidly review missedmessages and catch up to the current message (i.e., the live message);(vi) no dedicated circuit is needed for the conversation to take place,as required with conventional phone calls; and (vii) lastly, to initiatea conversation, one can simply begin transmitting to an individual or agroup. If the person or persons on the other end notice that they arereceiving a message, they have the option of reviewing and conducting aconversation in real time, or reviewing at a later time of their choice.

The communication media management system also supports new modes ofoptimizing the transmission of data over a network. The system activelymanages the delivery of payloads to a recipient engaged in aconversation in real time when network conditions are less than ideal.For example when network conditions are poor, the system intentionallyreduces the quality of the data for transmission to the point where itis “good enough” to be rendered upon receipt by the recipient, allowingthe real time participation of the conversation. The system alsoguarantees the eventual delivery of an “exact” copy of the messages overtime. The system and method therefore provides the advantages of bothspeed and accuracy. The utilization of network bandwidth is optimized bymaking tradeoffs between timeliness and media quality, using thepresence and intentions of whether or not recipient(s) intend to reviewto the message immediately in real time, as well as measures of networklatency, network degradation, packet loss or damage, and/or currentbandwidth conditions.

It should be noted that the messages of conversations may contain voiceonly or voice, video and other data, such as sensor information. Whenthe messages are reviewed, they are listened to or visually reviewed, ora combination thereof, depending on the type of media contained in themessages. Although as of the filing of the present application, mostconversations are voice only, it is intended that the communicationsystem and method described herein broadly includes conversationsincluding multiple media types, such as voice and video for example.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings, whichillustrate specific embodiments of the invention.

FIG. 1 is a diagram of the architecture of the communication and mediamanagement system of the invention.

FIGS. 2A and 2B illustrate a block diagram of a Client running on aDevice in the communication and management system of the invention.

FIG. 3 is a block diagram of a Server used in the communication andmedia management system of the invention.

FIGS. 4A through 4D illustrate various embodiments of data payloads usedin the communication and management system of the invention.

FIG. 5 is a diagram illustrating data being transmitted over a shared IPnetwork in accordance with the invention.

FIG. 6 is a diagram illustrating data being transmitted over a circuitbased network in accordance with the invention.

FIG. 7 is a diagram illustrating data being transmitted across both acellular network and the Internet in accordance with the invention.

FIGS. 8A through 8F are a series of flow diagrams illustrating a storeand stream function of the communication and management system of theinvention.

FIGS. 9A through 9C are flow diagrams illustrating the operation of aPayload Quality Manager (PQM) and FIGS. 9D through 9F are flow diagramsillustrating the Data Quality manager (DQM), both used by the Clientsand Servers of the invention.

FIG. 10 is an exemplary device having a graphical user interface thatmay be used with the system of the invention.

FIGS. 11A through 11F are diagrams illustrating multiple conversationmanagement (MCMS) features of the invention.

FIGS. 12A through 12C are diagrams illustrating the multipleconversation management system—consecutive (MCMS-C) features of theinvention.

FIGS. 13A through 13D illustrate a series of diagrams detailing theoperation of the invention.

FIGS. 14A and 14B are block diagrams that illustrate the hardware usedfor running the Client and Server applications of the invention.

It should be noted that like reference numbers refer to like elements inthe figures.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

The invention will now be described in detail with reference to variousembodiments thereof as illustrated in the accompanying drawings. In thefollowing description, specific details are set forth in order toprovide a thorough understanding of the invention. It will be apparent,however, to one skilled in the art, that the invention may be practicedwithout using some of the implementation details set forth herein. Itshould also be understood that well known operations have not beendescribed in detail in order to not unnecessarily obscure the invention.

A. Functional Overview

An improved voice and other media communication application, method andapparatus is disclosed which provides one or more of the followingfeatures and functions:

-   i. enabling users to participate in multiple conversation types,    including live phone calls, conference calls, voice messaging,    consecutive or simultaneous communications;-   ii. enabling users to review the messages of conversations in either    a live mode or a time-shifted mode (voice messaging);-   iii. enabling users to seamlessly transition a conversation between    a synchronous “live” mode and a time shifted mode;-   iv. enabling users to participate in conversations without waiting    for a connection to be established with another participant or the    network. This attribute allows users to begin conversations,    participate in conversations, and review previously received    time-shifted messages of conversations even when there is no network    available, when the network is of poor quality, or other    participants are unavailable;-   v. enabling the system to save media payload data at the sender and,    after network transmission, saving the media payload data at all    receivers;-   vi. enabling the system to organize messages by threading them    sequentially into semantically meaningful conversations in which    each message can be identified and tied to a given participant in a    given conversation;-   vii. enabling users to manage each conversation with a set of user    controlled functions, such as reviewing “live”, pausing or time    shifting the conversation until it is convenient to review,    replaying in a variety of modes (e.g., playing faster, catching up    to live, jump to the head of the conversation) and methods for    managing conversations (archiving, tagging, searching, and    retrieving from archives);-   viii. enabling the system to manage and share presence data with all    conversation participants, including online status, intentions with    respect to reviewing any given message in either the live or    time-shifted mode, current attention to messages, rendering methods,    and network conditions between the sender and receiver;-   ix. enabling users to manage multiple conversations at the same    time, where either (a) one conversation is current and all others    are paused; (b) multiple conversations are rendered consecutively,    such as but not limited to tactical communications; or (c) multiple    conversations are active and simultaneously rendered, such as in a    stock exchange or trading floor environment.-   x. enabling users to store all conversations, and if desired,    persistently archive them in a tangible medium, providing an asset    that can be organized indexed, searched, transcribed, translated    and/or reviewed as needed;-   xi. enabling the system to provide real time call functionality    using a best-efforts mode of message delivery at a rate “good    enough” for rendering as soon as possible (similar to UDP), and the    guaranteed eventual delivery of exact copies of the messages as    transmitted by requesting retransmission of any missing or defective    data from the originally saved perfect copy (similar to TCP); and-   xii. enabling the system to optimize the utilization of network    bandwidth by making tradeoffs between timeliness and media quality,    using the presence and intentions of the recipient(s) (i.e., to    either review the media in real-time or in a time-shifted mode), as    well as measures of network latency, network degradation, packet    loss or damage, and/or current bandwidth conditions.

In various embodiments, some or all of the numerous features andfunctions listed above may be implemented. It should be understood,however, that different embodiments of the invention need notincorporate all of the above listed features and functions.

B. Glossary

Prior to explaining the details of the invention, it is useful to definesome of the terms and acronyms used throughout the written description.This glossary of terms is organized into groups of System Components,Media, Media Management, People and Conversation Management.

B.1. System Components

Client: A Client is the user application in the communication system,which includes a user interface, persistent data storage, and “Voxing”functionality. Users interact with the Client application, and theClient application manages all communications (messages and signals) andpayload (Media) transfers that are transmitted or received over anetwork. The Client supports encoding of media (e.g., the capturing ofvoice, video, or other data content) and the rendering of media andsupports security, encryption and authentication as well as theoptimization of the transmission of data across the network. A Clientmay be used by one or multiple Users (i.e., multi-tenant).

Device: A physical device that runs the Client application. A User maybe actively logged into a single Device or multiple Devices at any givenpoint of time. In various embodiments, a Device may be a general-purposecomputer, a portable computing device, a programmable phone, aprogrammable radio, or any other programmable communication device.

Servers: A computer node on the communication network. Servers areresponsible for routing Messages sent back and forth between Users overthe network and the persistent storage and archiving of Media payloads.Servers provide routing, transcoding, security, encryption andauthentication and the optimization of the transmission of data acrossthe network.

B.2. Media

Message: An individual unit of communication from one User to another.Each Message consists of some sort of Media, such as voice or video.Each Message is assigned certain attributes, including: (i) the Usersending the message; (ii) the Conversation it belongs to; (iii) anoptional or user created Importance Tag; (iv) a time stamp; and (v) theMedia payload.

Media: Audio, video, text, position, sensor readings such astemperature, or other data.

Conversation: A thread of Messages (identified, persistently stored,grouped, and prioritized) between two or more Users on their Devices.Users generally participate in a Conversation using their Devices byeither Reviewing Messages in real time or in a time-shifted mode, orcreating and sending Messages of a Conversation as desired. When newMessages are created, they either define a new Conversation, or they areadded to an existing Conversation.

Head of a Conversation: The most recent Message of a conversation thathas been encoded by the most recent speaker. It is where a User ispositioned in a Conversation when reviewing “live” or where one jumps toif the “Jump To Live” feature is used.

Multiple Conversation Management System or MCMS: An application thatruns as part of a Client application, which enables a User to engage inmultiple Conversations using a variety of Media types. With the MCMSapplication, a User selects one Conversation among the multipleConversations as current, where only the Messages of currentconversation are rendered. For the selected current Conversation, theUser may transition from a series of back and forth Messages intime-shifted mode to near-synchronous “live” mode, similar to a standardtelephone conversation, and back again. The Messages of the non-selectedConversations are in a paused state. Messages associated with thenon-selected Conversion will accumulate if others are stillparticipating in those Conversations. The User may selectivelytransition the current Conversation among the multiple Conversations andReview the accumulated Messages of the selected current Conversation.

Multiple Conversation Management System-Consecutive or MCMS-C: Similarto MCMS, with the added feature of rendering and enabling Users tomanage and participate in multiple Conversations consecutively through ahierarchical system of Priorities and time-shifting, which areautomatically managed by the system. The MCMS-C application allows theMessages of consecutive Conversations to be rendered in a prioritizedorder, as opposed to MCMS where only the Messages of the currentlyselected Conversation are rendered. MCMS-C is particularly applicable insituations where it is important that the Messages of the consecutiveConversations are rendered, in the prioritized order, and/or the receiptof all Messages, even those belonging to lower priority Conversations,is more important than receiving the Messages in real-time. Examples ofsituations where MCMS-C may be suitable include, but are not limited to,hospitals, taxi fleet management, or tactical communications.

Multiple Conversation Management System-Simultaneous or MCMS-S: Similarto MCMS, with the added feature of enabling With MCMS-S, multipleConversations are selected for simultaneous rendering, as opposed toMCMS where the Messages of only the selected current Conversation arerendered. The MCMS-S application is particularly applicable insituations where a User is listening to multiple Conversations at thesame time, such as a trader listening to multiple brokers on differentexchanges and periodically sending trading requests to one or multipleof them simultaneously. MCMS-S may also be suitable for tacticalcommunications as well.

Priority: The mechanism through which the system determines whichMessage to render next when a User is participating in MCMS-C. Priorityis automatically managed by the system. A User can set defaultPriorities, or a predetermined set of system Priorities may be used. Inthe event of a conflict, where more than one Message is ready to berendered at the same time, the system resolves the conflict at leastpartly based on Priority, to determine what Message to renderimmediately and what Message to time shift.

Tags: a set of attributes a User or the system may assign to aConversation or a message, such as a topic (a company name), a directive(“action items”), a indicator (“conversation summary”), or any otherlabel by which one might want to search or organize the data.

Importance Tags: A special Message attribute that enables a sender tospecify when a Message is to be rendered, regardless of other Prioritysettings. An “urgent” Importance tag will override other Priorities forexample. This feature is critical for tactical systems, though anysystem can be configured to use or disable this feature.

Packet: Any unit of binary data capable of being routed through acomputer network. Each packet consists of header (meta data) and payload(media data). Includes standard packet protocols such as, but notlimited to, Internet Protocol (IP), EvDO, UMTS or any other packet-basednetwork, either radio, fiber optic, or wired.

Header or Packet Header: The portion of a packet that describes thepacket; the meta data concerning the payload, its encoding type anddestination.

Vox packet: A proprietary packet that enables the system and method tofurther refine and optimize the delivery of Messages, Media and othersignaling information.

Media Payload (or Payload): The actual Media portion of a Packet.

B.3. Media Management

Time Shift Delay (TSD): The amount of time between the arrival of a VoxPacket and the rendering of the Packet on a Device. The TSD must exceedthe Minimum Time Shift Delay. The TSD is typically determined by theUser's behavior in choosing to review the Messages of a Conversationsome time after receipt.

Minimum Time Shift Delay (MTSD): The time shift delay enforced by theClient to allow for jitter processing using jitter buffer techniques.This causes the system to delay rendering until an adequate number ofthe packets have arrived to create a usable media stream. The systemwill typically adaptively adjust the MTSD over time to compensate forvariable conditions in the network.

Rendering: Delivering a Media stream to a User in a form suitable forUser consumption (e.g., voice, text, graphic display, video, or acombination thereof).

Mixing: The Rendering of one or more Media streams. For example, theMedia stream from two Participants of a Conversation may be Mixed whenRendered, creating a User experience similar to a conversation wheremultiple people are speaking at the same time.

Encoding: The process of translating Media either created by a User(such as voice or video) or otherwise originating on a Device (such asGPS or other sensor data), and converting the media into digital data tobe processed by a Client.

Adaptive Jitter Buffer: Jitter buffers or de-jitter buffers are used tocounter jitter (i.e., either the arrival of out of sequence packets orthe delayed arrival of packets) introduced by packet switched networks,so that the continuous rendering of audio (or video) signals transmittedover a network can be performed without disruption. The data is storedin a buffer before Rendering to allow a reasonably sized buffer of Mediato arrive. The Media may be rendered before all the Packets arereceived, trading off quality for currency. An Adaptive Jitter Buffer iscapable of dynamically changing its size to optimize the delay/qualitytradeoff.

Persistent Infinite Message Buffer (PIMB): The PIMB is a storagemanagement system for the storage of time-based Media that performs boththe de-jittering of “live” data and the storage and retrieval of archivedata. The PIMB further includes the additional attributes of potentiallyinfinite and persistence storage of Media. The PIMB maintains “exact” orfull copies of Vox Packets of a Message and Conversations at some or allParticipant Devices and/or Servers.

Packet Loss Compensation or Concealment) (PLC): During Rendering of aMedia stream, the PLC component compensates for missing Packets,interpolating the results to present the stream to a reviewer. MissingPackets may be rendered as silence, or information from adjacent Packetsmay be used to present an interpolated sound or image. The particularmethod to be used will be dependent on the media, Codecs in use, andother generally known parameters.

B.4. People

User: A person who is authorized to use the system.

Contact: A record of either a User or non-user of the system. Userstypically engage in Conversations with members on their list ofContacts. A non-user is a user that accesses or uses the system using alegacy phone, radio or other non-Client 12 enabled device.

Group: The association of multiple Contacts. Contacts may be selectivelyadded or deleted from a Group. When a Conversation takes place among aGroup, all the members of the Group may or may not participate.

Channel: Typically used for tactical communication systems. A Channel issimilar to a Group in that it associates multiple Contacts with theChannel

Participant: A person who is identified as a member of a Conversation.Could be a User or a non-User participant.

B.5. Conversation Management

Time Shifting: Time shifting is the ability to play any Message at anytime after it has been received as determined by the User-recipient. ByTime-Shifting, a User may Review a Message: (i) immediately on demand byRendering immediately after the MTSD; or (ii) time-shifted in a mode ofreviewing the Message upon the discretion of the User; (iii) from thearchive for searching, reconstructions, etc. of old Conversations; (iv)after a delayed period of time to accommodate the Reviewing of otherhigher Priority Messages (or Conversations) that need to reviewed first;(v) and/or repeatedly if necessary for the Message to be reheard andunderstood. In other words, Time Shifting is the ability of a user torender a Message at any time after the system imposed MTSD.

Reviewing: Listening, viewing, reading or otherwise observing the Mediacontent in Messages. Reviewing may take place in either a nearsynchronous real-time “live mode” or the time-shifted mode.

Intention: Either (i) a User-defined attribute capturing whether theUser wants to Review the Messages of a Conversation either as soon aspossible or Review the Messages in a time-shifted mode; (ii) implied bya User's behavior; or a combination of (i) and (ii). .

Attention: A user attribute capturing whether the User is Reviewing theMessages of a given Conversation at the moment.

Catch Up To Live (CTL): A rendering mode that allows a User, who is notat the Head of a Conversation, to Review previous Messages more quicklyto “Catch Up To Live” (i.e., the Head of the Conversation). The CTLfeature may use any of a number of catch up techniques, such as thefaster replay of Messages, the removal of gaps in the Media of theMessages, removal of hesitation particles, etc. When the User has caughtup to live, the system seamlessly flows into the live Conversation. Thisis a very useful feature with conference calls, for example, insituations where a User needs to temporarily shift their attention awayfrom the Conversation, but wishes to hear the entire Conversation upontheir return.

Catch Up Mode: A user-configured or pre-configured mode that determineshow the CTL process will catch-up (i.e., replay faster, remove silence,and hesitation particles, or a combination thereof).

Jump To Live (JTL): This feature allows a user to jump from theircurrent position to the Head of a Conversation. A user will typicallyuse the JTL feature when they do not want to Review all of the Messagesbetween their current position in the Conversation and the Head. Whenthe JTL feature is implemented, the user skips over any interveningMessages and starts Rendering the “live” Messages at the head of theConversation.

MCMS Participant Attributes: A set of attributes, either defined by aUser, interpreted by the system from the User's behaviors, assigned byan administrator, or a combination thereof, which define the Intention,Attention, Priority, and rendering preferences of a receiver for a givenConversation. The attributes include, but are not limited to: (i) theIntention of when a receiver would like to render to the Messages of theConversation. Possible Intention values include: “now”, “time-shifted”,“Catch Up To Live” (CTL), “paused”, and “never”; (ii) Catch Up Mode,which is a configuration setting which determines how the CTL processshould catch the receiver up to live (e.g., play faster, skip silencegaps or hesitations, or play at normal speed); (iii) Time Shift Delay(TSD), which defines how far the receiver's current position in theconversation is from the Head of the Conversation, and (iv) the Priorityof the Message with regard to the receiver's other Conversations.

C. System Architecture

Referring to FIG. 1, a block diagram of the telecommunication and mediamanagement system according to one embodiment of the invention is shown.The system 10 includes a plurality of Clients 12 ₁ through 12 _(n),running on Devices 13 ₁ through 13 _(n) respectively. The Devices 13communicate with one another over a communication services network 14,including one or more Servers 16. One or more networks 18 ₁ through 18_(n), is provided to couple the plurality of Devices 13 ₁ through 13_(n) to the communication services network 14. In various embodiments,the networks 18 may be the Public Switched Telephone Network (PSTN), acellular network based on CDMA or GSM for example, the Internet, atactical radio network, or any other communication network, or acombination thereof. The communication services network 14 is a networklayer on top of or otherwise in communication with the various networks18 ₁ through 18 _(n). In various embodiments, the network layer 14 iseither heterogeneous or homogeneous. Clients 12 ₁ through 12 _(n)communicate with one another and with Servers 16 over the networks 18 ₁through 18 _(n) and network 14 using individual message units referredto as “Vox packets”, which are described in detail below.

D. Client Architecture

Referring to FIGS. 2A and 2B, a block diagram of a Client 12 running ona Device 13 is illustrated. As illustrated in FIG. 2A, the Client 12includes Multiple Conversation Management System (MCMS) application 20,a rendering and encoding module 21, and an MCMS applications database22. As illustrated in FIG. 2B, the Client 12 further includes a Storeand Stream (SAS) module 24 with a Persistent Infinite Message Buffer(PIMB) reader 26, a PIMB writer 28, PIMB database 30, a data and networkquality (DNQS) store 32, and Media driver and encoder hardware 34. TheMCMS application 20 and the Store and Stream module 24 communicate withone another through message handling modules 25 a and 25 b respectively.The Client 12 further includes an authentication-encryption-securitymodule 40 and a communication protocol module 44.

The module 40 provides authentication, encryption and security servicesduring the transmission and receipt of “Vox” packets to and from theClient 12. The communication protocol module 44 encapsulates Vox packetsinto the native packets used by the underlying network 18 connected tothe Device 13 running the Client 12 when transmitting data andde-encapsulating Vox packets from the native packets when receivingdata. With the modules 40 and 44, multi-party end-to-end authentication,encryption and security is provided between Clients 12. Messages areauthenticated, encrypted and secured across the networks 18 ₁ through 18_(n) and network 14, from a first sending Device 13 to second receivingDevice 13.

D.1.1. The MCMS Database

The database 22 stores and manages the persistent meta data for a numberof entities in the system 10, including Contacts and Participants,Conversations and Messages (live and stored), and default Priorities,and information regarding the Servers 16. In addition, the MCMS database22 stores the moment-to-moment operational data of a User'sConversations, presence, and status, as well as that of all theParticipants conversing with the User or on the User's Contact list. Forexample, with regard to Conversations and Messages, the database 22keeps track of status information, such as what Messages of aConversation a User has or has not Reviewed, Priorities, and Catch Up ToLive status for each Conversation in which the Client 12 is aParticipant, the presence and status of all Participants, and othernetwork and other system management data.

D.1.2. The MCMS Application

MCMS application 20 supports the different Voxing modes of engaging inconversations and/or managing multiple conversations using a variety ofmedia and data types (voice, video, text, location, data, etc.). Usersengage in Conversations by sending Messages to designated recipientsusing their Client 12 enabled Devices 13. Depending on preferences andPriorities, the recipient might Review the Message in real time, or theymight simply be notified that the Message is ready for Reviewing. Userscan transition from a series of back and forth Messages, which areReviewed in a time-shifted (or voice messaging) mode or in a nearsynchronous, full duplex conversation (similar to standard “live” phonecalls) and then back to voice messaging again. The MCMS application 20allows a User to control their interactions with their most importantConversations in real-time without missing any Messages in other ongoingConversations. For example, the MCMS application 20 notifies a User ofurgent or high priority communications from a Conversation that they arenot currently Reviewing. MCMS application 20 also enables all Messagesfrom all Conversations to be stored for later retrieval so they can bereviewed at any time.

In accordance with various embodiments, there are several differentoperational modes of the MCMS application 20, including MCMS-Consecutive(MCMS-C) and MCMS-Simultaneous (MCMS-S), which support the consecutiveand simultaneous rendering of Messages respectively. Each of theseembodiments is described in more detail below. Unless specificallyspecified, the term “MCMS” is intended to generally mean the MCMSapplication 20, which includes the aforementioned different modes.

The MCMS application 20 is a multi-tiered architecture that includes anumber of modules and services. The modules and services include theMCMS Database Module 20 a, the SAS Services Module 20 b, the Messagingand Signaling Services Module 20 c, the User Interface ApplicationProgramming Interface (API) 20 d, the User Interface Module 20 e, theConversations/Messages Management Services 20 f, the Priorities Services20 g, the Contacts Service 20 h, the Presence/Status Services 20 i, andthe Messages/Signals Services 20 j.

D.1.2.1 The MCMS Database Module

The MCMS database module 20 a is a service module that manages allfunction calls necessary for the MCMS application 20 to access the MCMSdatabase 22.

D.1.2.2 The SAS Services Module

The SAS Services module 20 b includes a set of function calls thatenable communication and coordination between the MCMS application 20and the Store and Stream module 24, and which are passed back and forththrough the message handling modules 25 a and 25 b respectively. The setof function calls enable both the MCMS application 20 and the Store andStream module 24 to operate as necessary to implement the various Voxingfunctions when invoked by Users and/or as dictated by networkconditions. Some of the functionality performed by the SAS Servicesmodule 20 b includes maintaining and communicating the status of Messagetransmissions and Message acknowledgments, the instructions forrendering of Messages, and the status and presence of Users.

D.1.2.3 The Messaging and Signaling Services Module

The Messaging and Signaling Services module 20 c runs on both Clients 12and Servers 16 and enables communication between the Client 12 and theServers 16 of the system 10. This communication, which includesmessages, data and other signals, allows the Client 12 and the system 10to track and administer communications, network status, Users, and Userstatus. Types of messages and signals sent between the Message andSignaling Services modules 20 c running on the Clients 12 and theServers 16 include, for example, network availability of Users, trackingof Messages that the Server 16 has sent to the Client 12 (possiblyincluding a “high water mark”) to determine if an entire message or someportion of a message is missing, (e.g., a sequence number perParticipant per Conversation created by the “generating” Client),whether a user is speaking or Reviewing Messages of a givenConversation, where a User is with respect to the Head of aConversation, or when a Participant is no longer Reviewing aConversation live. These are examples a few of the many types ofmessages and signals sent between the Message and Signaling Servicesmodules on the Clients 12 and Servers 16 and in no way should beconstrued as limiting the invention.

D.1.2.4 The User Interface API

The User Interface API 20 d is a module that defines a set of functioncalls that define the programming interface between the User Interfacemodule 20 e and the underlying services of the MCMS application 20. TheUser Interface API 20 d supports general-purpose methods such as UIapplication support, and all function calls necessary for a UserInterface to operate the MCMS Application 20. In various embodiments,the User Interface API 20 d enables the Client 12 to support a widevariety of user interfaces and device types, such as Adobe Flash-basedand/or Microsoft Windows applications, cellular or mobile phone devices,PSTN devices driven with tones, a voice user interface (VUI), andphysical radio communication interfaces. In various embodiments, theUser Interface API module 20 d enables the design of both highlyflexible and highly constrained user interfaces to support thefunctionality of the MCMS application 20.

D.1.2.5 The MCMS User Interface Module

The MCMS User Interface module 20 e supports the operation and functionsof the audio and video user interface of the Client 12. The UserInterface module 20 e supports a host of user interactions and can beimplemented with various interaction mediums, such as, an array ofgraphical user interface screens, an Audio/DTMF interface, or voice userinterface on the Device 13, all of which enable a User to interface withthe system 10. A partial list of User interactions that are supportedinclude, for example, functions to: log-in; manage, join, and monitorConversations; control Conversation rendering; manage Priorities; andrequests to review archived Conversations. It should be noted that thislist is exemplary and in no way should be construed as limiting theinvention.

D.1.2.6 Conversation/Message Management Services

The Conversation/Message management services 20 f is a module whichdefines a set of functions that manage the data structures and processesresponsible for managing and retaining all information needed for theUser to manage the receipt and Review of transmitted and received Media(e.g., voice or video content Messages) between the participants of aConversation. The Messages are organized into Conversations. Media thatis sent or received by the Device 13 running the application 12 isavailable for immediate Review while being received. The received Mediais also recorded for Review in a time-shifted mode, Conversationmanagement, and archival purposes. In an alternative embodiment,Messages or Conversations can be optionally marked for transience,specifying their desired retention requirements (e.g., some Messageswill not be retained or stored beyond the requirements for immediaterendering). In yet another embodiment, Media can be optionally markedfor review in a time-shifted mode only and cannot be reviewedimmediately upon receipt.

The Conversation/Message management services 20 f further enables, foreach current or ongoing Conversation of the User, the sending of Mediato a receiving Client 12 at any time, and the receiving Client 12seamlessly associates these Messages with the appropriate Conversation,regardless of the actions or inaction of the receiver.

With the Conversation/Message management services 20 f, allConversations are essentially asynchronous. If two Users are activelyengaged in a given Conversation and the User controlled delay betweentransmissions is minimal, the experience will be one of a synchronousfull duplex conversation, as with current telephone or VoIPconversations. If either User delays their participation, for whateverreason, the Conversation drifts towards an asynchronous voice (or otherMedia) messaging experience. In alternative embodiments, Conversationscan be optionally Tagged as asynchronous Messages only or synchronousMessages only. In either of these cases, the Conversation cannot driftbetween the two modes, unless the Tag is reset. After the Tag is reset,the Conversation again may flow between near synchronous (i.e. live orreal-time) and asynchronous (i.e., time-shifted or voice messaging)modes.

The Conversation/Message management service 20 f processes thetransmission and receipt of Messages in a progressive fashion. Whentransmitting, Media may be created while Messages are simultaneouslyencoded, stored and transmitted. In other words, the transmission ofMessages may occur simultaneously with the generation of Media by theUser (i.e., while speaking into their Device 13 or generating video). Onthe receiving side, the receipt, storage, and Rendering of Messages alsoall occur progressively. Messages do not need to be completely receivedbefore they can be Rendered. The Rendering of Messages may occur at thesame time Messages are being delivered, right up to the MTSD. Further,the service 20 f is also capable of the simultaneous transmission ofoutgoing Messages and Rendering of incoming Messages. The progressivenature of the service 20 f allows a User to be engaged in a liveConversation while storing and streaming the media of the Conversationfor later retrieval and review as well other functions described herein.

The time-shifting of Messages by the Conversation/Message managementservices 20 f allows a User to “Catch Up To Live” on a Conversation ifthey missed earlier Messages or were involved in another Conversation.This time-shifting process eliminates the need for Users to broadcast arequest to their entire Group or Channel to have Messages repeated.Older Messages may be replayed at any time at potentially higher speedsto save time. Users may easily skip forward and backward through theirMessages and within individual Messages. The Reviewing process may beconfigured on a Message-Priority basis to potentially skip lowerpriority Messages.

In one embodiment, the Conversation/Message management service 20 f alsoidentifies Messages by a specific Participant (speaker) and, by default,mixes Messages of a Conversation that were delivered at the same time(MCMS-S). In an optional embodiment, a User could Review thetransmissions of different Participant speakers of a Conversationseparately.

The Conversation/Message management module 20 f further allowsConversation sharing among Participants, who can be added to an activeor archived Conversation. In one embodiment, an added Participant to aConversation is provided access to the Conversation and has the abilityto retrieve the previous Messages of the Conversation for Review. In analternative embodiment, the added Participant is provided access to theMessages of the Conversation only from the point where the newParticipant joined, and not any previous Messages of the Conversation.

The Conversation/Message management module 20 f is also responsible formanaging the functions used to control all rendering tasks performed bythe Store and Stream module 24. These tasks include rendering Media(i.e., voice, video, etc.) appropriately for the Device 13 runningapplication 12. These rendering tasks include, but are not limited to,the rendering of Mixed Messages (i.e., overlapping messages), as well asrendering according to user-defined criteria, such as playing faster,catching up to live, removing silence, removing hesitation particles,frequency shifting, and the ability to apply independent gain control toindividual senders in a multi-party conversation.

D.1.2.7 Priority Services

The Priority service 20 g is a module that defines a set of functionsthat manage the data structures and processes responsible for managingand retaining all information needed for a User to manage the Priorityof the consecutive Conversations (i.e., MCMS-C) in which the User isengaged. When a User participates in a number of consecutive liveConversations, the User is required to prioritize the Conversations.Issues arise when Messages of different Conversations are ready to berendered at the same time. An algorithm is used to determine the orderin which the Messages are rendered that considers the availability ofMessages to be rendered and the Priorities set by the User. Thealgorithm determines that the available Messages with the highestpriority are rendered first while any concurrently available Messagesare time shifted automatically just enough to allow for the rendering ofthe higher priority Message. As rendering time becomes available, thesystem will automatically render the time-shifted messages according tothe User's Priorities.

D.1.2.8 Contacts Services

The Contacts services 20 h is a module that defines a set of functionsthat manage the data structures and processes responsible for managingand retaining all information needed for authenticating and associatingone or more Contacts with Conversations. When sending a Message as partof a Conversation that is associated with a number of Contacts, all ofthe Contacts receive the Message.

D.1.2.9 Presence/Status Services

The Presence/Status services 20 i is a module that defines a set offunctions that maintain the data structures and processes responsiblefor managing and sharing presence and status information between certainUsers and/or non-users of the system. In various embodiments, thepresence and status information is maintained for all User and non-usersengaged in the Conversations the User of the Client 12 is engaged in,all Users and non-users in the Contacts list, or Users within apredefined domain (e.g., the members of a corporation or otherorganization). These examples are merely illustrative and should not beconstrued as limiting. The Presence/Status services 20 i module maymanage and share presence and status information on any defined set ofUsers and/or non-users.

The Presence/Status service 20 i enables Users to monitor the status ofother User's Intentions, Attention, and their Time Shift delay on anygiven Conversation (i.e., how far back they are from Reviewing theMessages of the Conversation live). In one embodiment, privacy controlsare provided concerning availability of Presence and Status data. ThePresence/Status module 20 i further controls the data that enables thesystem 10 to deliver Messages that match the behavior and Intentions ofUsers. For example, a User may indicate their Status by designating anIntention to either Review or not Review a Conversation live. Inresponse, the Presence/Status services 20 i issues commands that causethe rendering of Messages either “live” or time-shifted, in accordancewith the Intention of the User. In addition, the Intentions of Users areshared with the other Participants of the Conversation. The service 20 iis also capable of inferring other Status values from the User'sbehaviors. The Presence and Status information is also used to optimizenetwork traffic and bandwidth, as described in more detail below.

D.1.2.10 Messages/Signals Services

The Messages/Signals services 20 j is a module that defines a set offunctions that manage the data structures and processes responsible formessaging and signaling Users of the system 10, using special messagesor audible tones. The special messages or tones may include for examplean indication if a Message or Messages are live or time-shifted, whomthe Message(s) are from, Priority, and other factors. The Message/Signalservice 20 j further has the ability to (i) signal the presence orabsence of Users on the network, as well as the ability to notify if oneor more Users are no longer actively Reviewing the Messages of aConversation; (ii) “ring” or otherwise notify another User to get theirattention when they are paying attention to another Conversation or notpaying attention to their Device 13 at all; (iii) leave a Message forUsers currently not on the network 18 to immediately review the Messagethe next time the individual re-connects to the network 18; (iv)generate an audible or visible feedback that alerts the sender that asent Message was not received by recipient(s), generate a confirmationwhen the Message has been received by the recipient(s), and/or aconfirmation indicating when the Message has been Listened to by therecipient(s); and (v) implement a Priority scheme where individuals on aConference or tactical call may be notified that their attention isimmediately needed on the call. This indication may convey multiplelevels of urgency and an acknowledgement of some kind by the recipient.

D.1.2.11 Rendering and Encoding

The rendering and encoding module 21 is responsible for performing allrendering tasks for the MCMS application 20. These tasks includerendering Media appropriately for the device 13 running application 12.

D.2 The Store and Stream Module

The Store and Stream module 24 supports a number of functions andperformance attributes, which are described below.

With the Store and Stream module 24, Message transmission is essentially“full-duplex”, enabling any party to send a Message at any time, evenwhile another party is also sending a Message, or if the other party isunavailable or otherwise engaged. The Store and Stream module is able torender messages as in a live PSTN or VoIP call or deliver them for timeshifted messaging modes. It is able to optimize transmission and controlRendering according to the desires of the User.

The Store and Stream module 24 maintains connectivity with all targetrecipients (e.g., Servers 16 or other Devices 13) on the underlyingnetwork 18, manages all message, signal, and media transmissions, andoptimizes the delivery speed and bandwidth usage across the network 18to meet a User's immediate performance requirements, while managingnetwork quality and capacity. The module 24 adapts and optimizes Mediadelivery commensurate with the quality and capacity of the underlyingnetwork 18. When insufficient underlying network resources areavailable, the quality of the transmitted Media streams can be degraded.As bandwidth becomes available, the quality of the transmitted Mediastreams may be increased. In addition to tradeoffs of Media quality, theStore and Stream functionality can make tradeoffs in the amount of Mediatransmitted in each packet based on Users' intentions to render data inreal time as described below.

By dynamically controlling the delivery rate of Media based on theconditions of the underlying network 18, the Store and Stream module 24is optimized to deliver time-sensitive Media that is “good enough” toRender upon receipt, and the guarantee eventual delivery of exact orfull copies of the Media for archival purposes through a backgroundprocess of requesting retransmission of missing, low quality, or damagedpackets. As long as sufficient network resources exist to meet minimumMedia quality levels, this retransmission does not impede the Renderingof live call Media. The Clients 12 of the system 10 are thus designed tobridge the performance gap between the delivery of an exact or completecopy of the Media at the expense of substantial potential latency versusthe quick delivery of Media, but with no guarantees of completeness. Inthe context of this application, the term “good enough” means that thequality of the Media is sufficient so that when it is rendered, it isintelligible. The notion of “good enough” is therefore subjective andshould not be construed in absolute terms. For example, the qualitylevel of certain Media to be good enough may vary depending on the typeof Media, circumstances, and other factors.

The Store and Stream module 24 further persistently stores all Mediacreated by or otherwise originating using a Device 13 or received overthe network 18 from other Device 13 and/or users. There are severalsignificant advantages of storing this Media on the Device 13 runningthe Client 12: (i) it enables Users to leave a Message for anotherparty, even when the sender and/or the recipient has either unavailableor poor network connectivity. In the case of insufficient bandwidth, theMessage will be transmitted as fast as available bandwidth can beeffectively used. In the case of no connectivity, the Message is queuedfor transmission as soon as network connectivity becomes available,resulting in a time-shifted delivery; (ii) the User has the ability topause, replay, fast-forward, and Catch-Up-To-Live with an ongoingConversation, as well as retrieve and review the archived Messages ofprevious Conversations; and (iii) it enables the optimization of datapayloads over the system 10 and improves system resilience againstnetwork bandwidth and connectivity problems that may occur from time totime.

The Store and Stream module 24 is also responsible for: Mixing Messagesas appropriate to create overlapping Messages (generated by the normaloverlap of speakers in a Conversation or background noise), simulatingan actual Conversation where multiple parties are speaking; renderingtranscriptions or translations of audio media; adjusting the renderingof Media according to a number of User-defined criteria includingplaying faster, removing silence gaps between spoken words, removinghesitation particles, and frequency shifting; and the ability to applyindependent gain control to individual senders in a multi-partyConversation; as well as other potential Rendering options.

The Store and Stream module 24 manages control and informationalmessaging between itself and MCMS.

D.2.1 The Persistent Infinite Message Buffer (PIMB)

The Persistent Infinite Message Buffer or PIMB 30 is a set of indexed(i.e., time-stamped and sequentially numbered) Media payload datastructures and a system for their storage and retrieval. In oneembodiment, the data in the PIMB 30 is arbitrarily persistent, meaningit is available virtually forever or at least until the system runs outof storage. Various retention rates and strategies may be employed tomake effective use of storage resources. Many possible implementationsexist for the physical storage implementation of the PIMB 30, including,but not limited to: RAM, Flash memory, hard drives, optical media, orsome combination thereof. The PIMB 30 is also “infinite” in size,meaning the amount of data that can be stored in the PIMB 30 is notinherently limited. This lack of limit is in comparison to existingjitter buffer technology that discards data as soon as it is rendered.In one specific embodiment, the PIMB 30 may be implemented using a smalland relatively fast RAM cache memory coupled with a hard drive forpersistent storage. As the physical storage capacity of the PIMB 30 isexceeded, the data is maintained on the Server 16 (as described below)for later retrieval on demand. User criteria or a replacement algorithm,such as least-recently-used, or first-in-last-out, is used to controlthe actual data stored in the PIMB 30 and the data that is maintained onthe Server 16 or archived at any point in time. The PIMB 30 furtherprovides the attributes of file system storage and the random accessattributes of a database. Any number of Conversations, regardless oftheir duration or the number of Messages in each, may be stored andlater retrieved for Review. In addition, the meta data associated withthe Messages of a Conversation, such as its originator and its length,may be also stored in the PIMB 30. In alternative embodiments, theindexed Media payloads and other data can be stored for a designatedperiod of time (e.g. 30 days). Once the age of the media exceeds thedesignated period, the payloads and data are discarded. In anotherembodiment, payloads can be discarded based on the sender and/or therecipient of the Message containing the payload, or the topic of theConversation or Messages associated with the payload. In yet otherembodiments, payloads and data may be marked for transience, meaning theMessages will not be stored in the PIMB 30 beyond the requirements forimmediate Rendering.

D.2.2 The Data and Network Quality Store

The data and network quality store (DNQS) 32 is a data store for storinginformation regarding Media payloads and Vox packets that are eitherread from or written to the PIMB 30.

D.2.3 The PIMB Writer

The PIMB writer 28 writes data to the PIMB 30 for two basic purposes.The PIMB writer 28 writes data from a Media capturing device (e.g., amicrophone or camera) on the Device 13 running the Client 12 (“EncodeReceive”). The PIMB writer 28 also writes data received over the network18 from other Clients 12 into the PIMB 30 (“Net Receive”).

D.2.3.1 Encode Receive

For capturing Media from the Device 13, the PIMB writer 28 includesEncoder Receiver 28 a and a Data Storer 28 c. When a User speaks intothe microphone or generates video images with the Device 13 for example,the hardware 34 receives the raw audio and/or video signals and providesthem to the Encoder Receiver 28 a, which encodes the signals intoindexed media payloads (hereafter sometimes simply referred to as“payload” or “payloads”). The Data Store 28 c stores the payloads intothe PIMB 30. Other types of Media, such as sensor data is converted intopayloads in a similar manner

D.2.3.2 Net Receive

For storing Media received over the network 18 into the PIMB 30, the NetReceive function of PIMB writer 28 includes a Network Receiver 28 d, aData Bufferer 28 e, a Data Storer 28 f, a Data Quality Manager 28 g, anda Network Quality Manager 28 h. The Network Receiver 28 d receives Voxpackets over the network 18. The Data Bufferer 28 e places the receivedVox packets into their proper sequence and prevents the Rendering of theincoming Vox packets for at least the Minimum Time Shift Delay (MTSD)delay. The Data Storer 28 f transforms the packets into indexed mediapayloads and stores the indexed media payloads in the PIMB 30. As thepayloads are stored, the Data Quality Manager (DQM) 28 g notes anymissing or defective packets. If packets are missing or defective, theDQM 28 g schedules a request for retransmission over the network 18. Thesending device replies by resending the missing or defective packets.Eventually these packets are converted into indexed media payloads andstored in the PIMB 30. By retrieving the missing or defective packets,an “exact” copy of a sender's Message is eventually stored in the PIMB30. The retransmission of missing and/or defective packets does notdelay the rendering of Messages in real time, provided that the packetsthat have been delivered are of “good enough” quality and quantity.Retransmission requests may be delayed by the DQM 28 g if insufficientnetwork resources are available to support both the new “live” data aswell as the retransmission.

D.2.4 The PIMB Reader

The PIMB reader 26 reads data from the PIMB 30 for two basic purposes.The PIMB reader 26 accesses the PIMB 30 when data is to be rendered(“Render”) for the local Client 12. Data is also read from the PIMB 30when data is to be transmitted (“Transmit”) by the Client 12 over thenetwork 18.

D.2.4.1 Render

For the rendering of messages on the Client 12, the PIMB reader 26includes a Data Prioritizer 26 a, a Data Retriever 26 b, a Packet LossCompensation/Interpolator (“PLC/Interpolator”) 26 c, a Data Mixer 26 dand a Data Renderer 26 e. The Prioritizer 26 a prioritizes the data tobe rendered by building an ordered queue of messages that couldpotentially be Rendered. It uses User configured Priority for therendering of consecutive Conversations (MCMS-C). In addition, the DataPrioritizer uses the availability of media data to render within thelimits imposed by the MTSD, the User's current Attention, and the User'sdefined and implied Intentions. The Data Retriever 26 b retrieves theprioritized indexed media payloads from the PIMB 30. ThePLC/Interpolator 26 c performs packet loss compensation andinterpolation on the retrieved payloads, using known packet losscompensation and interpolation algorithms The particular method to beused is dependent on the media Codecs in use, and other well-knownparameters. The Mixer 26 d is used to appropriately mix concurrent datastreams from multiple Messages of a single Conversation together. Forexample, if two or more Participants of a Conversation are speaking atthe same time, the Mixer 26 d mixes the Messages, creating the effect ofboth Participants speaking at the same time. In an alternativeembodiment, the User has the option of Reviewing the multiple streamsfrom one Participant at a time. If only one Participant in theConversation is speaking, the Mixer 26 d may simply pass the singleMessage stream, without performing any mixing. The Renderer 26 e takesthe data from the Mixer module 26 d and converts it into a form suitablefor the hardware driver 34. The hardware 34 then drives either thespeaker or video display of the Device 13, depending on the type ofMedia, creating voice, video or some other audible and/or visiblenotifier on the Device 13.

D.2.4.2 Transmit

To prepare messages for transmission from the Client 12 over a network18, the PIMB reader 26 includes a Data Prioritizer 26 f, a PacketQuality Manager (PQM) 26 g, a Data Retriever 26 h, Packetizer 26 i, aTransmitter 26 j and an Acknowledger 26 k. The Data Prioritizer 26 fprioritizes the Messages for transmission over the network 18. Priorityis determined using the MCMS Participant Attributes related to payloadsavailable for transmission, network connectivity and bandwidthconditions, and the Intentions of Users beyond the next hop to eitherRender live or time-shifted, and, in some embodiments, possibleoptimizations of transmission bundling where multiple packets to anygiven next network hop are available. The prioritized packets are thenoptimized, using the PQM 26 g, which assures the timely delivery of“good enough” data quality for live Messages, while minimizing real-timebandwidth, as described in more detail below. The Data Retriever 26 hretrieves the appropriate payloads from the PIMB 30. The Packetizer 26 iassembles the payloads into Vox Packets, which are then transmitted bythe Transmitter module 26 j over the network 18. When the recipientreceives the Vox packets, an acknowledgement is sent back toAcknowledger 26k over the network 18 for notifying the sending User thata Message has arrived at its destination.

D.2.5 The Packet Quality Manager

The PQM 26 g has several optimization goals: (i) the timely delivery ofan adequate copy of time-sensitive Media (“as soon as possible to begood enough” for Rendering); (ii) the efficient use of availablebandwidth, meaning using the optimal transmission frequency, payloadquality, and packet size for the underlying network; and (iii) theability to dynamically adjust or make changes in the transmissionfrequency, payload content, payload quality, packet size, etc. asnetwork conditions change.

D.2.6 The Network Quality Manager

On the receiving side of a network transmission is the Network QualityManager 28 h (NQM). The NQM is responsible for observing specificproperties of network performance for each sender that has sent media tothe Client 12, comparing expectations of jitter, loss, and throughput totheir actual values. This is used to compute a Network Quality Rating(NQR) for every sender. This NQR is used to indicate sender availabilityand Conversation live-ness to the User of the receiving device.

D.2.7 The Data Quality Manager

The Data Quality Manager 28 g measures the quality of the data beingreceived over the network by observing packet loss, jitter, andthroughput. The DQM 28 g uses these measurements for three purposes: (i)to send receipt reports back to the sender; (ii) optionally using thosereceipt reports to request retransmission of certain data; and (iii)making these measurements available to the NQM 28 h.

E. Server Architecture

Referring to FIG. 3, a block diagram of the application 78 that runs onthe Server(s) 16. The application 78 is similar to the Clientapplication 12 in many regards and includes an MCMS server application80, an MCMS database 82, a store and stream module 84, a PIMB 85, a dataand network quality store (DNQS) 86, MCMS-SAS message handling modules87 a and 87 b which manage messages and signals back and forth betweenthe MCMS server application 80 and Store and Stream module 84, anarchive/retriever 88, and an archive 89. The application 78 furtherincludes an authentication-encryption-security module 40 and acommunication protocol module 44.

The MCMS server application 80 is a multi-tiered architecture includinga MCMS Database Module 20 a, a Store and Stream (SAS) Module 20 b, aMessaging/Signaling Module 20 c, Conversations/Messages managementservices 20 f, Priority services 20 g, Contacts (including User andAuthentication) services 20 h, Presence/Status service 20 i, andMessages/Signals services 20. The aforementioned modules and services ofthe application 78 are similar or the same as the modules and serviceshaving like reference numerals as the Client 12, and therefore are notdescribed in detail herein, except for one notable exception. In variousembodiments, the MCMS server application 80 and Store and Stream module84, including the MCMS database 82, is configured to support many Usersin one instance of the application. The one instance may be furtherconfigured to support multiple domains, where each domain is defined asa group of Users (i.e., a corporation or other group of Users belongingto a common organization). This architecture allows each application 78on Server 16 to serve multiple users (or domains), where each user (ordomain) is not visible to other Users. This partitioning is referred toas “multi-tenancy”.

The server Store and Steam 84 module performs the functions of NetReceive and Transmit. For the Net Receive function, the module 84includes a Net Receiver 28 d, Data Bufferer 28 e, a Data Storer 28 f, aData Quality Manager (DQM) 28 g, and a Network Quality Manager 28 h. ForTransmit functions, the module 84 includes a Data Prioritizer 26 f,Packet Optimizer 26 g, Data Retriever 26 h, Packetizer 26 i, Transmitter26 j and an Acknowledger 26 k. The aforementioned elements of the Storeand Stream module 84 are similar or the same elements having likereference numerals as the Client 12, and therefore are not described indetail herein.

Since the Server 16 does not directly interact with Users, the encodingand rendering functions provided in the Store and Stream module 24 ofthe Clients 12 need not be present. The MCMS application 80, whenrunning on Servers 16, does not interact directly with Users.Consequently, the user interface and user interface API modules andservices 20 e and 20 d are not needed.

The application 78 on each Server 16 potentially serves multipletenants, meaning it serves multiple Users of the system 10. The PIMB 85of the server application 78 is therefore significantly larger and isused to store the media payloads of multiple Users, as opposed to thePIMB 30, which is used to store just the generated or received payloadsof only a single User. The main purpose of the Store and Stream module84 is to receive Messages transmitted by the Clients 12 and transmitMessages to other Clients 12. When a Client 12 transmits a message, themessage contains an identifier associated with the target recipient. Inresponse, the Server 16 finds a delivery route to a Client 12 associatedwith the intended or targeted recipient. As Messages are received, theyare stored in the PIMB 85 and transmitted to the next Server 16 (i.e.,the net “hop”) of the network layer 14 along the path to the intendedrecipient(s), or to the recipient(s) directly depending on the systemconfiguration. The archive-retriever 88 is responsible for archiving themedia payloads stored in the PIMB 85 in archive 89. As the physicalspace in the PIMB 85 is exhausted, media payloads in the PIMB 85 aremoved to the archive 89, which is a mass storage device. In variousembodiments, the payloads stored in the PIMB 85 may be archived inaccordance with User defined criteria and/or any known replacementalgorithm, such as first-in-first-out (FIFO) or least recently used(LRU). It should be noted that only a single Server 16 is illustrated inFIG. 1 for simplicity. It should be understood that in actualembodiments, multiple servers or a “server farm” may be used for anetwork with a large number of Users.

The terms “persistent” and “infinite” used to describe the PIMB 30 andthe PIMB 85 should not be construed literally as absolute terms. A Usermay wish to indefinitely store some Messages that are consideredimportant. In other situations, such as a casual chat between twofriends, the Messages may be deleted after a certain period of time tosave space. According to various embodiments of the invention, differentretention policies may be used, either set by the system 10 orconfigured by the User. The use of the word “infinite” refers to thelack of any preset time boundaries enforced by the PIMB. This iscontrasted with current jitter buffer systems, which discard media afterit has been rendered. The terms persistent and infinite should thereforebe broadly construed to mean that the PIMB 30 and PIMB 85 have nointernal limitations on the time ranges and quantities of Messages thatcan be stored therein.

There are a number of advantages to archiving the Messages ofConversations in a persistent storage medium. Voice messages and otherMedia can be organized, indexed, searched, transcribed, translated, andReviewed as needed. Voice, as well as other Media, therefore becomes anasset that can be managed both by Users and organizations. These Mediaassets have value to corporations, first responders, police and firedepartments, as well as the military.

F. The Vox Protocol and Indexed Media Payloads

As noted above, the Vox protocol is used by the Store and Stream module24 to support all facets of payload transmission, storage, andoptimization. The Vox packet is a structured message format designed forencapsulation inside a transport packet or transport packets of theunderlying technology of the network 18. This arrangement significantlyimproves the flexibility of the system 10. By embedding the Vox packetsinto existing transport packets, as opposed to defining a new transportlayer for “Voxing” applications, the system 10 takes advantage ofcurrent packet based communication networks running over the existingtelecommunications infrastructure. A new network infrastructure forhandling the Vox packets therefore need not be created to take advantageof all the benefits of the system and method described herein.

Referring to FIG. 4A, the general format structure of a Vox packet 95 isillustrated. The format of the Vox packet 95 includes fields for type,sub-type, length, and payload. The different types of Vox packetsinclude authentication, signaling, media payload, media multiplex (onemessage), and media multiplex (multiple messages). The sub-type field isused to designate different types of authentication, signaling or mediatype messages. Possible sub-types for authentication Messages includethose necessary for key exchanges and Authentication. Possible sub-typesfor signaling Messages include registration, routing, message set-up,and network management. Possible sub-types for Media messages includedifferent Codec styles and different payload aggregation techniques. Thelength field defines the overall length or size of the payload. Thepayload field contains the actual payload or media of the packet 95.

Referring to FIG. 4B, a diagram illustrating a Vox packet 95encapsulated in an exemplary protocol used by the network 18 is shown.In this example, the Vox packet 95 is embedded in underlying UDP, IP andEthernet transport packets 96 respectively. In this manner, the Voxpacket 95 can be transported across underlying UDP, IP and Ethernetlayers of the network 18. This is a standard protocol encapsulationtechnique used by packet networks.

Referring to FIG. 4C, a diagram illustrating a media multiplex Voxpacket 95 encapsulated in UDP, IP, and Ethernet 97 is illustrated. Inthis example, the Vox packet 95 includes a Media type field, a Mediasub-type field, a length field, a message ID field, a time stamp field,a sequence ID field, and a Media payload field.

Referring to FIG. 4D, the format of an indexed media payload 98 isillustrated. The indexed media payload includes a sub-type field, alength field, a message identifier (ID) field, a time-stamp field, asequence identifier (ID) field, and field for the Media payload.

The encapsulation of Vox packets 95 into the transport packets of theunderlying network allows the Media, Messages and Conversations to eachbe defined by a number of attributes.

When Media is created or otherwise originated on a Device 13, it istypically time-based, meaning it changes in some meaningful way overtime. As a person engages in a Conversation for example, their spokenwords are strung together into sentences or statements, which vary overtime, and the resulting data (streams and packets) will maintain thesame variance over time. Similarly video (as opposed to a still photo)as well as GPS or other sensor data will vary over time. Regardless ofthe type or how it is originated, the Media is segmented and placed intothe payloads of a plurality of Vox packets 95. The packets are thencontinually, stored, transmitted, received, stored and rendered instreams (i.e., streaming media) at the transmitting and receivingDevices 13 respectively. Since each packet 95 is indexed, time-stamped,and given a sequence identifier, the individual packets can be segmentedinto Messages. By sequentially threading the individual Messagestogether, Conversations may be constructed.

One further unique aspect of the system 10 is that the media payloadsgenerated by a Client 12 are stored in multiple locations. Not only arethe payloads stored in the PIMB 30 of the generating Device 13, but alsoin the PIMB 85 of the Server(s) 16 and the PIMB 30 of the receivingDevices 13. This basic feature enables or makes possible much of theVoxing functionality described above and provides the system 10 withboth resilience and operability, even when network conditions are pooror when a Participant of a Conversation is not connected to the network.

G. Interoperability With Underlying Telecommunication Protocols

The system 10 is intended to run or be layered over a variety ofexisting communication networks 18, such as the Internet, fixed PSTNtype circuit networks, and mobile or cellular phone networks, or acombination thereof. The system 10 is designed around the concept ofmoving many small units of information (i.e., the Vox packets) betweendifferent Clients 12 and Servers 16 in the system 10. While Vox packetsmay vary in size, depending on their function and payload, they allappear to be the same kind of data to the underlying network layer. Inone embodiment, the system 10 has been designed and optimized for IPv4networks such as the Internet, but other types of networks may besupported as well. For the purposes of this document, the term “IP”should be taken to mean IPv4, IPv6 or any other current or futureimplementation of the Internet Protocol.

Referring to FIG. 5, a diagram of a Client 12 running on Device 13 andcommunicating with a Server 16 over a shared IP network 100 is shown. Asillustrated, the Client 12 is coupled to the shared IP network 100through a first Internet service provider A and the Server 16 is coupledto the shared IP network 100 by a second Internet service provider B.During communication, the Vox packets 95 (designed “VP” in the figure)are encapsulated within UDP/IP packets and then interleaved among otherIP protocol packets as is well known in the art and transmitted acrossthe shared IP network 100 from the Client 12 to Server 16, or viceversa. As is well known, each lower packet layer encapsulates the entirepacket of the layer immediately above it. Packets can also be sent in asimilar manner between two Servers 16. Messages are sent from one Client12 enabled Device 13 to another over a shared IP network 100. At eachhop, the Vox packets 95 are embedded in the underlying IP protocol andtransmitted, until they reach the target destination.

The diagram of FIG. 5 is merely exemplary, showing only a single Client12 and Server 16 connected to the network 100 for the sake ofillustration. In actual embodiments of the system 10, a large number ofClients 12 and one or more Servers 16 are typically connected to theshared IP network 100. It is also useful to note that the Client 12 andServer 16 do not have exclusive use of the IP network 100. In theexample shown, an HTTP client, which is coupled to the network 100through Internet provider A, can send packets back and forth with anHTTP server, coupled to the network 100 through a third Internetprovider C. The system 10 does not control the manner in which the VPsembedded in the IP packets traverse the network 100. Rather, all packetsthat traverse and share the network 100 do so in accordance with thestandard procedures of the underlying shared IP network 100.

Referring to FIG. 6, a “circuit” based network 104 such as a GSM mobilephone network is illustrated. The circuit network 104 is coupled betweenClient 12 running on Device 13 and Server 16. Once a circuit isestablished between the Client 12 and Server 16, the system 10 layersVox packets (VP1, VP2, VP3, VP4, VP5, etc.) onto the underlying packetsused by the network 104 and transmits them across the network 104,creating “virtual Vox” circuit. The Vox Packets sequentially traversethe circuit network 104, typically with spacing or framing data as iswell known in the art for transmitting data over a circuit network. Inaddition, packet construction parameters, such as the payload size andthe number of header fields included may be used to exploit the lack ofper-packet overhead and to increase speed and/or efficiency of datatransfer across the network 104. It should be noted again that for thesake of simplicity, only a single Client 12 and Server 16 are shownconnected to the network 104. It should be understood, however, thatadditional circuits between Clients 12 and Servers 16 as well as othercomponents may be established concurrently through the network 104. Thenetwork 104 is therefore not dedicated for the transmission of VoxPackets, but rather may be shared with other types of network traffic.

Referring to FIG. 7, a diagram illustrating communication between afirst Client 12A enabled Device 13A associated with a first network Aand a second Client 12B enabled Device 13B associated with a secondnetwork B is illustrated. The networks A and B further each includegateway Servers 16A and 16B respectively. The Gateway Server pair 16Aand 16B facilitate communication between the two networks A and B,allowing the Devices 13A and 13B to communicate with each other. Invarious embodiments, the networks A and B could each be any type ofnetwork. For example, each network A and/or B could be an IP network, acircuit type network, or a wireless or cellular network (i.e., CDMA,GSM, TDMA, etc.). The Servers 16 that straddle the two networks A and Bare considered gateway servers because they route traffic or serve as a“gate” between the two networks.

With the system 10, there are a several basic network interactionconsiderations to optimize system performance. These considerationsinclude factors such as resolving the underlying address to which theVox packets 95 are to be sent, the integrity of any sent Vox packets,and the management of the Maximum Transmission Unit (MTU) of a singleMessage that may be sent across a given network or combination ofnetworks. The address of a target Client 12 needs to be known so thatthe underlying network delivers the Vox packet 95 to the correctlocation. With IPv4 networks, the address is typically an IPv4 Address,which is a 32-bit number that uniquely identifies a host within thenetwork. For other networking technologies, the address could be someother type of identifier. IP networks use the Domain Name System (DNS)to resolve human-readable names into IP addresses, and the AddressResolution Protocol (ARP) to resolve IP addresses into physicaladdresses. Regardless of the underlying networking technology, thesystem 10 uses one of the above-mentioned or other known addressingschemes for delivery of Vox packets 95 to the correct location.

As with almost any packet-based communication system, transmitted Voxpackets might not be delivered to their addressed location if theunderlying network is unable to deliver the packets in which the Voxpackets are encapsulated. Most packet networks do not informtransmitters when packets are dropped. Instead they rely on thetransmitters and receivers to notice and compensate for any droppedpackets. The system 10 is designed to use these receiver receipt reportmessages to coordinate this packet loss management. If the underlyingnetwork is able to inform the sender of lost or dropped packets, thesystem 10 utilizes this information in its retransmission protocol.

The management of MTU is the determination of the Maximum TransmissionUnit (i.e., the maximum size of a single message) that may be sentacross a network. For packet-based networks, the underlying networkimposes the MTU. For circuit-switched networks, the MTU may be a tunableparameter for network efficiency and performance. Thus in most cases,the underlying network imposes or determines the maximum size of the Voxpacket 95 that may be transmitted efficiently. For example with IPnetworks, packets may be fragmented if the payload exceeds the MTU, butat a substantial performance penalty. With IP over Ethernet networks,the transmitting device has an MTU of 1518 bytes, as enforced byEthernet. The largest IP packet must leave room for the Ethernetheaders. The largest UDP packet must leave room for both IP and Ethernetheaders and the largest Vox protocol that may be generated on Ethernetfor example is the Ethernet MTU (1518)-IP header (20)-UDP header(8)=1490 bytes. Since the Vox protocol will have a header of its own,the actual Vox media payload will be less than 1490 bytes on an Ethernetnetwork. For Gigabit Ethernet, the MTU could be much larger, but wouldbe determined using a similar formula.

In a purely packet-based network, there are two potential values forMTU, the local link MTU and the path MTU. Determining the local link MTUyields the maximum size for Vox packets to be efficiently sent out tothe local network interface. The path MTU yields the maximum size of Voxpacket that may be sent intact all the way to the remote node. If asender is connected via Ethernet, the Vox packet might pass throughvarious other systems with smaller MTUs en-route to the recipient. Thesmallest MTU on the path to the destination needs to be resolved andknown by the sender. In the IP world, there is a standard procedure fordiscovering the smallest MTU called “Path MTU Discovery”. For otherkinds of networks, an equivalent procedure may be used. Again, since thesystem 10 is layered on top of other networks, any of the above MTUalgorithms may be used.

H. Operation Flow Diagrams H.1 Store and Stream

Referring to FIGS. 8A through 8F, a series of flow diagrams are providedto illustrate the operation of the store and stream module 24 and 84 onClients 12 and Servers 16 respectively. FIG. 8A shows the sequence ofoperation for a first Client 12 ₁ transmitting Messages to a secondClient 12 ₂. FIGS. 8B and 8C illustrate the operation of the PIMB writer28 and PIMB Reader 26 on the transmitting Client 12 ₁. FIGS. 8D and 8Eillustrate the operation of the PIMB Writer 28 and PIMB Reader 26 on thereceiving Client 12 ₂. FIG. 8F illustrates a flow diagram of the Storeand Steam module 84 on a server 16.

In FIG. 8A, the User of Client 12 ₁ running on a Device 13 ₁ originatesMedia to be transmitted. The Media can be originated at the Device 13 ina number of different ways such the User creating Media by speaking intoa microphone or creating video content on their Device 13. Media is alsooriginated by a Device 13 by receiving sensor data, such a GPSinformation or a temperature reading. Regardless of how the Media isoriginated, the Media is encoded by the PIMB Writer 28 (box 130), whichconverts the Media into indexed media payloads and stores them in thePIMB 30 (box 132) on Client 12 ₁. The PIMB Reader 26 on the Client 12 ₁reads the payloads out of the PIMB 30, creates Vox packets, andtransmits the packets to the receiving Client 12 ₂ (box 134) over thenetwork 18. Each Server 16 along the path between the sending Client 12₁ and the receiving Client 12 ₂ stores the transmitted payloads in thePIMB 85 and transmits the Vox packets to the next hop (box 133). At thereceiving Client 12 ₂, the net receive function of the PIMB Writer 28converts the Vox packets into indexed media payloads (box 136) andstores the payloads into the PIMB 30 of Client 12 ₂ (box 138). Therendering module of the PIMB reader 26 on Client 12 ₂ renders thepayload information read from the PIMB 30 into a medium suitable forhuman consumption, such as voice or video (box 140). Each of these stepsare described in more detail below with respect to FIGS. 8B through 8E.

In FIG. 8B, a sequence of the Encode Receive function performed by thePIMB Writer 28 (step 130 of FIG. 8A) is provided in detail. In theinitial step 130 ₁, the User of the Device 13 running the Client 12 ₁originates the Media to be transmitted. As noted above, the Media may bederived by speaking into a microphone, using a camera, receiving sensordata, or by some other Media generating component. In the next step 130₂, the Encode Receiver 28 a encodes the Media and creates the indexedmedia payloads (step 130 ₃), which are then stored in the PIMB 30 (step132) by the Data storer 28 c.

In FIG. 8C, the sequence of the Transmit function performed by the PIMBReader 26 (step 134 of FIG. 8A) on the sending client 12 ₁ is providedin detail. In decision loop 134 ₁, the transmit function of the PIMBReader 26 continuously checks to see if indexed media payloads that areto be transmitted have been written into the PIMB 30 and are availablefor transmission. If such payloads are available in the PIMB 30, theData Prioritizer 26 f prioritizes the payloads that should be sentfirst, using the MCMS Participant Attributes information, as illustratedin step 134 ₂. Information about the highest priority payloads arepassed to the Packet Optimizer module 26 g which runs the PQM (step 134₃), as described in more detail below with respect to FIGS. 9A-9C. Theappropriate payloads are then retrieved from the PIMB 30 (step 134 ₄) bythe Data Retriever 26 h and converted into Vox packets 95 by thePacketizer 26 i (step 134 ₅). The Vox packets 95 are then transmitted(step 134 ₆) by the Transmitter 26 j over the network 18 to the receiveClient 12 ₂, which sends back receipt reports reflecting the properties(loss, jitter, throughput) of the packets that have been received. Thesereceipt reports provide the information necessary for the PQM tocalculate the MABR for a given receiver. The aforementioned process isrepeated for each transmission loop as indicated by the return arrowfrom the transmit step to the top of the flow chart.

In the embodiment described above, the media is encoded, stored in thePIMB 30 and then transmitted over the network in a serial fashion. In analternative embodiment, the encoded media can be stored in the PIMB 30and transmitted over the network in parallel, meaning the two functionsoccur substantially at the same time.

In FIG. 8D, the sequence for the Net Receive function (step 136 of FIG.8A) of the PIMB Writer 28 on the receiving Client 12 ₂ is illustrated.In the initial step 136 ₁, the network receiver 28 d receives the Voxpackets 95 over the network 18. The Data Storer 28 f converts thepackets into indexed media payloads (step 136 ₃), which are stored inthe PIMB 30 (step 136 ₄). As the payloads are stored, the Data QualityManager (DQM) 28 g is run. The DQM 28 g checks for missing or corruptedpackets, ensures the eventually storage of an exact copy of thetransmitted data, and the sends receipt reports regarding the conditionsof the network to the transmitter. Each of these functions of the DQM 28g are described in more detail below with regard to FIGS. 9D through 9F.

In FIG. 8E, the sequence for the Render function of the PIMB Reader 26(box 140 of FIG. 8A) on the receive Client 12 ₂ is illustrated. In theinitial step 140 ₁, the Data Prioritizer 26 a prioritizes the indexedmedia payloads to be rendered as determined by the MCMS application 20using the MTSD information as well as User status and presenceinformation, including the User's Intentions and Attention status. Theprioritized payloads are then read from the PIMB 30 (step 140 ₂) by theData Retriever 26 b. The PLC/Interpolator 26 c performs packet losscompensation and interpolation (step 140 ₃) on the retrieved payloads,using known packet loss compensation and interpolation algorithmsdepending on which Codecs are used. In the next step, the Mixer 26 dmixes (step 140 ₄) multiple Messages of a Conversation if two or moreParticipants have generated Media at the same time within the sameConversation (e.g., both are speaking at the same time). The Renderer 26e renders (step 140 ₅) the data stream from the Mixer 26 d, generatingsound, video, or other Media (step 140 ₆) for the recipient User.

In FIG. 8F, the sequence for a Server 16 to receive Vox packets from theprevious hop on the network 18, store, archive and transmit the Voxpackets to the next hop is illustrated. In the initial step, the Server16 performs the Net Receive function of the PIMB writer (similar to FIG.8D) to store the indexed media payloads of the received data in the PIMB85 and archive 89 or the Server 16. The server 16 also performs theTransmit function of the PIMB writer (similar to FIG. 8C) to forward thereceived packets onto the next hop on the network 18. In this manner, acopy of the media generated by the transmit Client 12 ₁ is received,stored and transmitted at each hop along the path to the receive Client12 ₂.

In the aforementioned embodiment, the writing of received indexed mediais stored in the PIMB 91 of the Server 16 and transmitted to the nexthop in a serial fashion. In an alternative embodiment, the receivedindexed media payloads can be stored in the PIMB 91 and transmitted tothe next hop substantially at the same time. The storage of Media on thePIMB 30 of both transmitting and receiving Devices 13 allows for theprogressive transmission and rendering of Media. On the transmit side,the Media originating on the transmitting device may be progressivelytransmitted over the network as it is being received. In variousembodiments, the encoded Media (regardless of how it is originated) maybe progressively transmitted before, after, or at substantially the sametime it is stored in the PIMB 30. On the receive side, the incomingMedia may also be progressively rendered as it is received over thenetwork, provided the User has opted to Review the Media in the nearreal-time mode. In various embodiments, the incoming Media may beprogressively rendered before, after or substantially at the same timeas it is stored in the PIMB 30 of the receiving Device 13. If thereceived Media is to be Reviewed in the time-shifted mode, then theMedia is retrieved from the PIMB 30 (or possibly a PIMB 85 on a Server16 if replaced on the local PIMB 30) for later Review at a timedesignated by the User.

In the context of the present application, the term progressive orprogressively is intended to be broadly construed and generally mean thecontinuous processing of a data stream based on availability of thedata. For example, as Media is created or otherwise originated on aDevice 13, the progressive encoding, storage, packetization andtransmitting of that media is continuous, so long as the Media isavailable. As a person speaks, that Media is progressive or continuouslyencoded, stored, packetized and transmitted for the duration of thepersons speech. When the person pauses or stops speaking, there is nomedia to progressively process. When the person resumes speaking again,the progressive processing of the Media resumes. On the receive side,the Media is also progressive processed as the Media is being received(i.e., available). As the Media is received it is continuously stored.It will also be continually rendered as the Media is being received whenin the near real-time mode or from storage when in the time-shiftedmode. Although the above explanation was provided in the context ofvoice, it should be understood that all types of Media can beprogressively processed in a similar manner A1so the progressiveprocessing of Media does not necessarily have to be progressivelyprocessed in time-indexed order. Rather the Media is processed in theorder in which it is received. If Media is received out of index order,in one embodiment, the Media is progressively processed in the order itwas received and then organized into the indexed sequence in the PIMB30. In an alternative embodiment, the received Media can be organizedinto its indexed sequence and then progressively rendered.

H.2 POM Operation Flow Diagrams

The PQM 26 g relies on a metric called the Maximum Available Bit Rate(MABR), which is a continually computed approximation of actualtransmission capacity or bandwidth (i.e., a measure of the capability ofthe network at a given point in time) between a sending and receivingnode pair. As instantaneous network conditions change, the MABR isupdated. Regular measurements of network throughput, packet loss, andjitter are considered in computing the MABR. In an alternativeembodiment, the MABR may also be manually set or limited based on timeof day, type of network, other conditions or parameters.

The PQM also considers the Intention of the recipient(s) to optimizetransmission for time-sensitivity. A transmission is consideredtime-sensitive if either (i) the Intention of the recipient(s) is toReview the transmission “live” or in the near real-time mode, or (ii)the recipient would like to immediately Review a Message that for somereason is not currently stored on their Device 13 (e.g., the Message waspreviously stored in the archive 89). The Intention of the recipient canbe either inferred by the behavior of the recipient or the recipient mayset or otherwise designate their Intention. On the other hand, atransmission is considered to be not time-sensitive if the Intention ofthe recipient is to Review the Message in the time-shifted mode. TheIntention of the recipient to Review the transmission in either the liveor time-shifted mode at least partially defines the “timelinessrequirements” of the transmission. In various other embodiments, factorssuch as the urgency or priority of transmissions may also be consideredin defining the timeliness requirement of the transmission.

The nodes in the network path between a sender and a receiving pair alsoneed to be consistent regarding the status of intentions of therecipients. If one target recipient indicates timeliness, meaning theywish to Review the transmission immediately or live, then all theintermediate nodes on the network along the sender-receiver path need tohave the same timeliness requirement, regardless of the requirements ofother recipients. The timeliness requirement of each of the intermediatenodes is therefore dependent on the receiving nodes the transmission isbeing sent to. This dependency is sometimes referred to as a “union ofrequirements” for target nodes in the network transmission path.

The PQM further considers an Ideal Bit Rate or “IBR” for each scheduledMessage payload transmission. For time-sensitive transmissions, the IBRis computed based on the packetization rate needed for substantiallyreal time or live communication (referred to herein as the Real Time BitRate or RTBR). With voice for example, a packetization rate of a packetevery 20 milliseconds containing 20 milliseconds of audio data isconsidered an acceptable IBR for conducting live conversations. The RTBRfor such a system in kilobits per second would be the size of 1 secondof audio payload data plus the size of all network headers that would begenerated for the transmission. For video media or a combination ofvoice and video, the RTBR will likely be substantially higher thansimply voice. For other types of media such as sensor or GPS positioningdata, the RTBR will likely be lower than that of voice. For nontime-sensitive transmissions, the IBR is set to a Maximum Efficiency BitRate (MEBR) to optimize the usage or efficiency of transmissions overthe network. The MEBR is calculated by adjusting the packetization rateto its largest possible value for the underlying network. If multiplemessages or payloads are to be sent between a sending and receivingpair, then an Aggregate IBR (AIBR) is considered for the transmission.

The PQM operates by sending data in a series of transmission loops foreach sending and receiving pair. The transmission loops for each sendingand receiving pair are independent. Any transmission on the network mayaffect the MABR of other sending-receiving pairs. Accordingly, the MABRis preferably continually computed for all recipients.

Referring to FIGS. 9A through 9C, flow charts illustrating the sequenceof operation of the PQM for a single sending and receiving pair isillustrated. In FIG. 9A, the steps in determining the MABR between thesingle sending and receiving pair are illustrated. In FIG. 9B, a flowchart illustrating the steps for calculating the AIBR for eachtransmission loop for the single sending and receiving pair areillustrated. In FIG. 9C, a sequence for determining the amount of datato transmit between the sending and receiving pair per loop isillustrated. The processes illustrated in the three diagrams runsimultaneously and interact with one another, as described in moredetail below.

In FIG. 9A, a flow chart 50 for calculating the MABR for the networkinterface between the sending and receiving pair is shown. In theinitial step 50 ₁, the network interface between the sending andreceiving pair is monitored. The sender periodically receives reports,which contain information regarding the status of the network connectionat the receiver in step 50 ₂. The reports include information regardingthe current status of data throughput 50 ₃, packet loss 50 ₄, and jitter50 ₅ as observed by the receiver at the network interface. In step 50 ₆,the MABR is calculated at the sender based on these observationscontained in the reports. By monitoring or observing the data in thesereports, the MABR value is continually adjusted based on current networkcapabilities or conditions between the sending and receiving pair. Asnetwork capabilities become more favorable for data transmission, theMABR will increase. If the network capabilities become less favorablefor transmission, the MABR will decrease, potentially all the way tozero for an unusable network. The reports are similar to the packet lossreports generated by nodes in TCP networks, but additionally includethroughput and jitter information as well.

If there are several network interfaces between the sending andreceiving pair, an MABR is calculated for each interface for which areceipt report is received. If no traffic has been recently sent on thenetwork, or no receipt reports have been received, the MABR may notreflect current network conditions. However, since receipt reports aregenerated continually by receivers while data is transmitted, thesender's MABR metrics will quickly converge to a more accurate value.

Referring to FIG. 9B, a flow chart 52 illustrating the steps forcalculating the AIBR for a transmission loop is illustrated. In theinitial step 52 ₁, the Messages with Media (by which we mean portions ofthe time indexed media that belongs to this Message) ready to betransmitted between the sending and receiving pair in the current loopare ascertained. A list of Messages with Media is then built 52 ₂. Foreach Message in the list 52 ₃, the time-sensitivity or timelinessrequirement of each Message is considered 52 ₄. If a particular Messageis not time-sensitive, then the IBR is set to the Maximum Efficiency BitRate (MEBR) 52 ₅. On the other hand, if a Message is time-sensitive,then the IBR is set to the Real Time Bit Rate (RTBR) 52 ₆. In the nextstep 52 ₇, the previously computed IBRs for each of the Messages in thelist are summed together, resulting in the Aggregate Ideal Bit Rate(AIBR) 52 ₈ for the transmission loop. As signified by the return arrow52 ₉, the above-described process is repeated for each transmission loopbetween the sending and receiving pair.

Referring to FIG. 9C, a flow chart 54 illustrating the sequence fordetermining the rate of data to transmit between the sending andreceiving pair per transmission loop is illustrated. In the initial step54 ₁, the MABR (as calculated in FIG. 9A) is compared to the AIBR (asdetermined in FIG. 9B) for the next transmission.

If the MABR is greater than or equal to the AIBR, then all the Messagesidentified as ready for transmission in the loop are packetized at theIBR rate 54 ₂ and transmitted 54 ₃.

On the other hand, if the MABR is less than the AIBR, then a series ofprocedures are applied to so that the PQM meets its goals of the timelydelivery of an adequate copy of the data, the efficient use of availablebandwidth, and/or adjustments to the payload content and quality, packetsize, and transmission rate to meet current network conditions.

In the initial step, the Messages in the list are reviewed for timesensitivity 54 ₄. If there are no time sensitive Messages, then the bitrate is reduced to the MABR 54 ₅, and the Messages are transmitted 54 ₃.

If the list includes time-sensitive Messages, the bit rate allocated forthe non time-sensitive Messages is reduced 54 ₆ until it meets the MABRlimits. If reducing the bit rate all the way to zero is insufficient tomeet the MABR, then these non time-sensitive Messages are removed fromthe list of Messages to be transmitted in the loop. If the bit rate hasbeen reduced so that it is less than or equal to the MABR, then theremaining Messages are Packetized and transmitted 54 ₃.

If the removal of non time-sensitive Messages was not sufficient to meetMABR, then another procedure involving the use of a lower quality Codec(or Codecs) for the remaining time-sensitive Messages 54 ₇ is used. Anattempt is made to transmit the payload data as fast as possible bysending fewer bits during the transmission loop. In other words, byreducing the quality of the payload, the transmission sends fewer bitsin a given period of time. In various embodiments, different Codec orCodecs, each having a different bit rate versus quality tradeoff, may beused. If the use of the lower quality Codec or Codecs is sufficient,meaning the MABR limit is met, then the Messages are transmitted 54 ₃.

If the use of lower quality Codec(s) still does not meet the MABR, thenthe packetization interval of the time-sensitive Messages is increased54 ₈. With this procedure, the header-to-payload ratio is increased,which lowers the overall bit rate but introduces latency (i.e., a delayin the delivery of the transmission to the recipient). If this procedureresults in the reduction of the AIBR to less than or equal to the MABR,then the transmission 54 ₃ occurs.

If after changing the packetization interval the MABR is still not met,then the bit rate may be progressively lowered 54 ₉ to be within theMABR limit. When the bit rate is lowered in this manner, time-sensitivemessages are sent at a rate that is not sufficient to maintain a liveconversation. Therefore, the Conversation is forced out of “live”. Ifthe network is down or conditions are very poor, it is possible that nodata transmission may occur. Again, the aforementioned sequence isrepeated for each transmission loop 54 ₁₀ between the sending andreceiving pair.

If there are multiple network interfaces between the sending andreceiving pair, the sequence described in relation to FIG. 9C isperformed for each interface for which receipt reports are available. Invarious embodiments, the sender may contain logic to distribute thetransmission load among the multiple interfaces. In different examples,the payloads can be sent only on one interface, while in otherembodiments, some or all of the interfaces may be used.

The aforementioned description pertains to any sending and receivingpair in the system 10. In most situations, the sending and receivingpair will include a Client 12, enabled Device 13 and Server 16, twoServers 16, a Server 16 and Client 12 enabled Device 13, or evenpossibly two Clients 12 respectively. If a sending node is transmittingto two (or more) receiving nodes at the same time, the above mentionedsequence as described in relation to FIGS. 9A-9C occurs concurrently foreach receiving-sending pair.

H.3 DOM Operation Flow Diagrams

The DQM 28 g determines if data received at the Client 12 is eithercorrupted or if there are missing packets. In addition, the DQM 28 g ofa receiving Client 12 generates of the receipt reports, which are sentback to the transmitting node on the network. The DQM 28 g also runs abackground process to ensure that an exact copy of transmitted data iseventually received and stored. These functions are described below inFIGS. 9D through 9F respectively.

Referring to FIG. 9D, a flow diagram illustrating the operation of theDQM 28 g for checking for missing and/or corrupted data is illustrated.In the initial step 56 ₁, the DQM 28 g checks for corrupted packetsusing well-known techniques, such as CRC or similar integrity checkingmechanisms. If a packet is corrupted, that packet is treated as missing56 ₂. The DQM 28 g next ascertains if any packets are missing 56 ₃. Ifan out of sequence packet is not received after a predetermined periodof time, it is assumed to be missing. The DQM 28 g notes any missing orcorrupted packets 56 ₄ in the DNQS 32. On the other hand if no corruptedor missing packets are detected, the DQM 28 g determines if the qualityof the received data was intentionally degraded by the sender 56 ₅ forthe purpose of saving bandwidth. The degraded quality is noted in theDNQS 32 56 ₆. Regardless if the quality of the received data is degradedor not, receipt information (e.g., a packet sequence number, time stamp,and the network address of the next node in the network the packet is tobe sent) of the data is stored 56 ₇ in the DNQS 32. The aforementionedprocess is continually repeated, as signified by the return arrow to thestart of the flow diagram.

As a result of the process detailed in FIG. 9D, information regardingthe receipt of non-degraded packets, the deficiency of degraded qualitypackets, and missing packets, are all stored in the DNQS 32. As Media isreceived, the DNQS 32 maintains up-to-date information regarding thestatus of the Media.

Referring to FIG. 9E, a flow diagram illustrating the operation of thereceipt report generator function of the DQM 28 g is illustrated. In theinitial step, the DNQS 32 is periodically scanned 58 ₁ to determine ifthere is any Media for which a receipt report needs to be generated 58₂. If the answer is no, then the above scanning process is repeated. Onthe other hand if Media is identified, then the process determines ifthe Media is time-sensitive 58 ₃, meaning either the User intends toReview the Media live or the user would like to immediately Review Mediathat is not stored on their Device 13.

If not time-sensitive, then the recipient informs the sender to set theretransmission priority (as defined below) to low 58 ₄. Iftime-sensitive, then the amount of packet loss is considered 58 ₅. Ifthe amount of packet loss is outside a usable quality range, theretransmission priority is set to high 58 ₆. As noted above, if theamount of packet loss is too large, the recipient may not be enabled toReview the Media upon receipt. If the quality is within an acceptablerange, meaning the quality of the transmission is sufficient that it canbe understood when rendered, then the priority for the sending of thereceipt report is set to low 58 ₄. Regardless if the recipient isReviewing upon receipt or not, a receipt report is sent 58 ₇, the DNQS32 is updated 58 ₈ and the Network Quality Manager (NQM) 28 h is updated58 ₉. The retransmission requests defined in step 58 ₄ is thereforeconditional based on time-sensitivity. The transmission request definedin step 58 ₆ is conditional on both time-sensitivity and quality.

The retransmission priority informs the PQM 26 g of the sender toproperly prioritize the transmission rate for the Media that requiresretransmission. For example when the retransmission priority is set tohigh, then the sending PQM 26 g of the sender should send anyretransmissions before any new Media. If the priority is low, the PQM 26g should send the retransmitted Media after any new Media.

The aforementioned process is continuously repeated so that receiptreports are generated as Media is received. If the sender does notreceive receipt reports in a timely manner, the PQM 26 g of the senderwill reduce the transmission rate, eventually stopping the transmissionif no receipt reports are received.

Referring to FIG. 9F, a flow diagram illustrating the sequence forrequesting of missing or degraded Media is illustrated. In the initialstep 60 ₁, the DNQS 32 is periodically scanned for missing or degradedMedia 60 ₂. If there is no missing or degraded Media, then the abovedefined scan is periodically repeated.

Media is considered missing if an out of sequence packet does not arriveafter a predetermined threshold period of time 60 ₃. If the packetarrives before the threshold, then it is no longer considered missing.On the other hand if the packet does not arrive after the threshold isexceed, then it is deemed missing. With missing packets, a low priorityrequest for retransmission is made 60 ₄ and the time of the request isnoted 60 ₅ in the DNQS 32. This process is repeated until the missingpacket is received. When the missing packet arrives and thecorresponding Media is available in the PIMB 30, the missing status ofthe Media is removed from the DNQS 32. The retransmission requestdefined in step 60 ₄ is therefore conditional based on whether the Mediais determined to be missing.

If degraded, the DQM 32 determines if the Media is part of a liveConversation 60 ₆. If not, a request for a full quality copy of thedegraded Media is made 60 ₇, the full quality Media is designated asmissing 60 ₈ and the request time is noted 60 ₉, in the DNQS 32. If theMedia is part of a live Conversation, then no action is immediatelytaken in order to preserve network bandwidth. When the Conversationtransitions out of the live mode, then the steps 60 ₇ through 60 ₉ areperformed to ensure that a full quality (i.e. an exact or perfect) copyof the degraded Media is eventually received. When the data becomesavailable in the PIMB 30 of the recipient Client 12, the degraded statusof the associated Media is removed from the DQNS 32. The transmissionrequest defined in step 60 ₇ is conditional on whether the Media is bothdegraded and not part of a live conversation.

The aforementioned process is continually repeated, as signified by thereturn arrows from 60 ₅ and 60 ₉ to the top of the flow diagram at 60 ₁.By repeating the process outlined in FIG. 9F, exact copies of alltransmitted Media is eventually stored in the PIMB 30 of the receivingDevice 13. In this manner, the storage of exact copies of transmittedMedia is guaranteed at the recipient Device 13.

I. Graphical User Interface

Referring to FIG. 10, an exemplary Device 13 running Client application12 is illustrated. The Device 13 includes a graphical user interfacedisplay 110, data entry buttons, keys, or keyboard 112, a microphone114, and a transducer 116 for converting electrical signals into sound,such as a speaker. The display 110 may also accept input as a touchscreen. Further, the display 110 and keyboard 112 may be combined usinga touch screen interface. As noted above, the Device 13 may be a varietyof different communication tools, such as a desktop computer, laptop orother mobile computers, personal digital assistants, a programmableland-line or cellular phone, or programmable radios, or just about anyother type of programmable communication device. The exemplary Device 13illustrated in the figure is intended to be “generic”, in the sense thatit is supposed to represent or encompass all of the communicationdevices listed above. In addition, the term “graphical” user interfaceshould not be construed as limiting. Other types of user interfaceswhich can be implemented on a Device 13 as well including Audio/DTMFinterfaces, Voice User Interfaces (VUI), radio switch interfaces, or acombination thereof, can all be used to implement the various functionsdescribed below. For the sake of simplicity, each of these types ofmethods by which a User may interact with their Device 13 aregenerically referred to as a “user interface”.

A1l Devices 13, regardless of their type, preferably have a userinterface that enables the user to operate the Device 13 and communicatewith other User's in the system 10. Although the user interface can bedesigned to have virtually an infinite number of different look and feelimplementations, there are certain functions and features that allDevices 13 should have in common. These functions and features arelisted below.

The user interface preferably includes one or more of the followingstatus indicators or flags: (i) battery indicator; (ii) connectivityindicator; (iii) clock; (iv) transmitter status; (v) transmission syncstatus; (vi) Reviewing status; (vii) Priority Messages needingattention; and (viii) missed Messages.

The user interface preferably includes the following functions, flagsand components to conduct and manage a single Conversation: (i) name ofa Conversation and/or list of Participants; (ii) Conversation status;(iii) Conversation type; (iv) Conversation duration; (v) time behind theHead of the Conversation; (vi) outstanding Messages; (vii)presence/status of Participants; (viii) meta data with navigation; (ix)Conversation attributes or designators; (x) Conversation set-upincluding title, scheduling, Participants, Conversation summary; and(xi) indicators showing which Participants have contributed Messages andwhich have Listened to or Reviewed messages.

The user interface also preferably includes, in addition to those listedimmediately above, the following functions, flags and components toconduct and manage multiple Conversations: (i) a name/identifier foreach Conversation; (ii) live/active or standing/inactive indicator;(iii) Review position, either at Head or Time shifted; (iv) Priority andother attributes; and (v) indicators of what portions of Conversationswere missed.

The user interface also preferably includes, a number of navigationfeatures, such as: (i) DVR style fast back/forward per conversation;(ii) instant messaging style personal message navigation; (iii)Conversation Time indicators; (iv) time scale shifting (i.e. zoomingbackward and forward through a Message or the Messages of aConversation); (v) changing Priority of Conversations; (vi) hanging up;and (vii) home.

The aforementioned functions and features may be implemented in avariety of ways, for example using a touch-screen graphical userinterface 110, or other input devices, such as the data entry buttons,keys, or keyboard 112, a mouse, by voice-activated commands, or acombination thereof. Again the functions and features listed above andhow they are implemented is not intended to be exhaustive. The variousmethods and technologies that could be used is so comprehensive, that itis not practical to list or discuss them all herein.

J. Conversations

The MCMS application 20 supports a number of different types ofConversations, such as a near real time or “live” calls where the delayfrom when a Participant speaks and the other Participant(s) hear thefirst Participant is very small, Conversations where Participantsexchange voice Messages back and forth with a longer delay betweenMessages, “live” conference calls involving multiple Users, standingConference calls at a regularly scheduled times, or configurablestructured call types such as a simultaneous briefing, whereParticipants each leave a Message briefing beforehand for others toReview to before everyone joins the live conference call. Yet anotherunique attribute of the MCMS application 20 is the ability for Users totransition between the different types of Conversations. For example,the Participants can seamlessly transition from a voice-messaging modeto a live call and back again. Or the Participants of a live conferencecall can transition to a voice-messaging mode and send updates or actionitems to each other after the conference call. While several exampleshave been noted, it should be understood that the system 10 is extremelyflexible and provides numerous options to transition between differenttypes of calls or Conversations and between multiple conversations. Byvarying the delay between Messages, the Participants effectively flowsbetween the type of Conversation that best suits their needs. The aboveexamples should therefore not be construed as limiting.

As noted above, Conversations consist of Messages maintained in theiroriginal context and sequence. A sent Message either belongs to anexisting Conversation or begins a new Conversation. A typicalConversation includes a set of Messages that are organized around adefined subject matter, topic, time, Group or Channel. For example,Conversations may involve a common Group of people such as the membersof a club, a company may have a standing Conversation at a fixed time,such as a weekly sales meeting, or friends may have ad-hoc Conversationson various topics, such as making dinner plans.

Each Conversation is defined by a set of attributes, including a name, alist of Participants, begin and end time, and state including at leastpending, active, or terminated. Other Conversation states are possiblein other embodiments. A User interacts with the MCMS application 20 ontheir Devices 13. In preferred embodiments, the interface allows a Userto organize Conversations by any of the various attributes.

The relationship between a Participant and a Conversation also hasattributes. These attributes include, but are not limited to, Priority,Status (states of participation in a Conversation). The values ofParticipant status include active, participation in more than oneConversation at a time, Reviewing a Conversation in a time-shifted mode,Catching Up To Live, passively participating (i.e., not activelyReviewing, but receiving high Priority Messages), standby, or ignoring aConversation (i.e., declined to participate or record the Conversation).

From a recipient's perspective, a User may select or define the relativepriority of Messages. For example, Messages from a person's boss wouldtypically be given a higher Priority than a social acquaintance. In someembodiments, a recipient has the ability to set their own Priorities. Inthe implementation of MCMS-C the User selects a subset of theirConversations to be rendered consecutively. The User then sets orderedPriorities for these Conversations. The system uses the priorities setby the user to order the Messages to be rendered. The aforementionedalgorithm queues the Messages to be rendered using User Priorities andinformation concerning Message data available (beyond MTSD).

In other embodiments, such as tactical communications, the recipient mayhave no or limited ability to set Priorities. For example, a firefighter may have no ability to lower the Priority of Messages from thefire chief. A sending-User, however, has the ability to send urgent orhighly important Messages. By Tagging a Message as urgent or anemergency, the Message is rendered as soon as possible at therecipient(s), overriding any Priority settings of the recipient. Anyconflicts among multiple emergency messages are resolved based on apredetermined priority scheme.

K. MCMS Operation

Referring to FIG. 11A, an organizational diagram 1100 grouping the majorfunctions of the MCMS application 20 is illustrated. The major functionsinclude account management 1102, conversation management 1104, aggregateconversation list management 1106, conversation participation 1108, callcontrols 1110, and contact management 1112. After registering andlogging into the system 10, the User may navigate through the userinterface of their Device 13 implementing the various managementfunctions, described in detail below. In some embodiments, thisfunctionality will provide a great deal of flexibility. In otherembodiments, such as tactical or communication radios, theimplementation of the user interface may be constrained with many of theuser functionality and options preconfigured to meet the utility of thedevice. The discussion here is exemplary and is not intended to be anexhaustive explanation of the MCMS functionality, but rather is intendedto provide just an overview of some of the MCMS attributes.

K.1 Account Management

Under the account management function 1102, a registered User may changecertain settings and preferences. A User may change their email address,password, name, phone number, phone password, call-in number, defaultand/or User defined rendering speeds, Tags, gain or volume levels forReviewing Messages, Catch Up to Live mode, etc. To make these changes,the User enters the appropriate information through the interface 110 oftheir Device 13. The MCMS application 20 responds by writing the updatedpreferences into the MCMS database 22.

K.2 Conversation Management

As illustrated in FIG. 11B, conversation management 1104 is a set offunctions that allow a User to view their aggregate Conversation lists,create a new Conversation, update the details of a Conversation, deletea Conversation, and close a Conversation. Each of these functions aredescribed below.

View Conversations 1104 a—For each Conversation, the MCMS application 20may provide the User the one or more of the following attributes: thename of the Conversation, the actual start time, the last activity,Tags, duration, and the list of Participants. For each Participant, thename and/or phone number, status (live, other call, in past,catch-up-mode, offline-reachable, offline-unavailable.)

Create a Conversation 1104 b—A User creates a Conversation through theinterface 110 by inputting a Conversation name, a list of Contacts, andan optional scheduled start time. If no start time is designated, it isassumed the start time is immediate. In response, the MCMS application20 creates a new Conversation in the database 22, associating recordsfor each Participant on the Contacts list. The MCMS application 20 alsocreates in the database 22 Participant records for each User on theContact list, allowing the caller to receive the presence information ofthe others on the Contact list. If the Conversation is scheduled, theMCMS application 20 starts the Conversation at the designated time.Otherwise, the Conversation starts right away.

Update Conversation Details 1104 c—The User may make changes to aConversation through the user interface 110. For example, Participantsmay be added or removed. Any change in status of the participants isupdated in the MCMS database 22.

Delete a Conversation 1104 d—A User may delete a specific Conversationfrom their list of Conversations through the interface 110. In response,the MCMS application 20 notes the change in the database 22, anddesignates the Conversation as deleted.

Close a Conversation 1104 e—A User may elect to terminate or close aConversation. In one embodiment, only the User that creates aConversation can elect to terminate that Conversation.

K.3 Aggregate Conversation List Management

As illustrated in FIG. 11C, the aggregate conversation list management1106 is a set of functions that allow a User to engage in multipleConversations (i.e., the User's aggregate conversation list). Theaggregate conversation list management functions allow a User, throughthe interface 110 on their device, to participate in one Conversation“live”, while participating in other Conversations in a time-shiftedmode.

Chose Conversation 1106 a—Through the interface 110, a User may selectone Conversation among the User's aggregate conversation list ascurrent. The Messages of the current Conversation may be rendered ineither the “live” or time-shifted modes. The User may switch the currentconversation among the aggregate conversation list from time-to-time.

Switch Conversations Modes1106 b—In an optional embodiment, a User maybe able to switch from the MCMS, MCMS-C and MCMS-S modes of operation.

K.4 Conversation Participation

As illustrated in FIG. 11D, conversation participation 1108 is a set offunctions that allow a User to start a Conversation, receive anotification to join a Conversation, obtain Conversation statusinformation, and hang up on a Conversation.

Start a Conversation 1108 a—After a Conversation has been created,either by the User through the interface 110 or the scheduler in theMCMS application, the status of each Participant is checked. If aParticipant is offline, then an effort is made to contact that person.If a Participant is online but engaged in another Conversation, then theMCMS application 20 notifies that Participant. The presence status ofall online Participants is updated in the database 22.

Receive a Notification 1108 b—The system may notify a User that theirattention has been requested on a Conversation through a graphic displayand/or audible notification via the user interface 110.

Conversation Status 1108 c—A user may request the status of aConversation through the interface 110 of their Device 13. In response,the MCMS application 20 assembles the status information stored in thedatabase 22 and presents the information to the User.

Conversation Pause 1108 d—Through user interface 110, a User may hang upor switch away from an active Conversation. In response, the MCMSapplication 20 updates the User's participant status for activeConversations in the database 22 and directs the Store and Stream module24 to remove the User from the Conversation.

K.5 Conversation Controls

As illustrated in FIG. 11E, conversation control 1110 is a set offunctions that allow a User to control their participation in aConversation. These functions allow a User to catch-up-to live, skip tothe Head, jump to a past location, pause, play faster and play slowerwhen Reviewing the Messages of a Conversation. Each of these functionsis triggered by the User through the interface 110 on the Device 13.

Catch-up-to-live 1110 a—A User may Catch Up To Live in an ongoingConversation using the “CTL” function. When this function is activated,the MCMS application 20 checks the last point in the Conversation theUser has reviewed and directs the Store and Stream module 24 to renderthe Messages not previously heard, using a faster than normal renderingoption designated by the User, and to seamlessly transition to live modewhen it reaches the head.

Jump to Head 1110 c—This function enables a User to jump to the Head ofa Conversation, skipping over any intervening Messages between thecurrent point of the User in the Conversation and the Head. Whenimplemented, the MCMS application 20 instructs the Store and Streammodule to immediately render the Messages at the Head of theConversation. (If the Head of the Conversation is currently live this iscalled Jump to Live (JTL).

Jump to past 1110 d—This function enables a User to jump back to aprevious Message or point in the Conversation, similar to a rewind orreplay function. When implemented, the MCMS application 20 instructs theStore and Stream module 24 to render Media starting from the rewindpoint.

Pause 1110 e—This function enables the User to pause the Reviewing ofthe Messages of a Conversation. In response, the Store and Stream module24 stops the rendering of Messages until another command is issued.

Play faster 1110 f—This function enables a User to render Messages morequickly. In response, the Store and Stream module 24 renders theMessages at a faster rate than normal. The rendering rate may be eitherspecified by the User or the User may select from a number of presetoptions.

Play slower 1110 g—This function enables a User to render Messages moreslowly. In response, the Store and Stream module 24 renders the Messagesat a slower rate than normal. The rendering rate may be either specifiedby the User or the User may select from a number of preset options.

K.6 Contact Management

As illustrated in FIG. 11F, the system 10 provides the User with a hostof functions for managing Contact lists and user Groups. These functionsinclude the adding, editing, deleting Contacts and user Groups. Each ofthese functions are implemented by a User though the interface of theirDevice 13. Any revisions or deletions in a User's Contact list or Grouplist is stored in the MCMS database 22.

Add a Contact 1112 a—This function enables a User to add a new Contactto their contact list. The new Contact can be either a registered Useror an external contact. Typically the name, phone number(s), type ofnumber (cell, office, home, computer, etc.), email address and otherpersonal information are provided for each contact record.

Edit a Contact 1112 b—This function enables a User to edit or update anexisting contact record.

Delete a Contact 1112 c—This function enables a User to remove or deletean existing contact record.

Search for a Contact 1112 d—This function enables a User to search for aspecific Contact in their contact list. The search may be conductedusing a number of criteria, such as name, phone number, most recentlycalled, most frequently called, Group, etc.

Get a Participant list 1112 e—This function enables a User to search andretrieve a list of Participants of a Conversation by a number ofdifferent search criteria, including for example by name, most recentoutgoing calls, most recent incoming calls, most frequently called, etc.

Authorize a caller to review status 1112 f—This function enables a firstUser to authorize other Users to view the first User's status.Non-authorized Users may not view the status of the first User.

Create a Group of Contacts 1112 g—This function enables a User toassociate a number of Contacts into a Group. When a User defines aGroup, the list of Contacts in that Group are stored in the MCMSdatabase 22.

Edit a Contact Group 1112 h—This function enables a User to edit aGroup, or update the contact information for a member of the Group.

Delete a Contact Group 1112 i—This function enables a User to delete aGroup.

L. MCMS Operation L.1 MCMS-C

As noted above, the MCMS-C operation is similar to MCMS, with the addedfeature of enabling Users to manage and participate in multipleconversations consecutively through a hierarchical system of Prioritiesand the time-shifting of Messages, which are automatically managed bythe system. Implementing the MCMS-C functionality includes three basicprocesses. As illustrated in FIG. 12A, the first process involvesdefining a set of Conversations for consecutive rendering. Once the listis defined, a hierarchical set of Priorities and other factors areapplied to the indexed media payloads associated with the set ofConversations. The indexed media payloads are then sequenced into asequencing order. By rendering the media in the sequence order, theMessages of the set of Conversations are consecutively rendered.

Referring to FIG. 12A, a flow chart illustrating the steps for definingthe list of Conversations to render consecutively is shown. In theinitial step 1202, the User's aggregate list of Conversations isdefined. Either the User or pre-configuration data (step 1204) is nextused to select the Conversations among the aggregate list forconsecutive rendering (step 1206). With a tactical communication systemfor example, typically pre-configuration data is used to impose theConversations to be consecutively rendered. With non-tacticalapplications, the User is typically given a higher degree of flexibilityto select the Conversations for consecutive rendering.

Referring to FIG. 12B, a flow chart illustrating the steps for defininghierarchical set of priorities for rendering the Messages of consecutiveConversations is illustrated. In the initial step (1208), a set ofpriority rules are defined and applied to the list of Conversations tobe rendered consecutively (1206). In various embodiments, the set ofpriority rules may range from a rigid, hierarchical communicationprotocol to a highly flexible communication protocol. For example in atactical communication system where a rigid hierarchy is oftendesirable, the set of priority rules will preferably impose a specificorder in which concurrent Messages are rendered. For example with afirst-responder tactical system, Messages from a fire chief may be giventhe highest priority. The next level of priority may be given to firefighters inside a burning building. At the next level, priority may begiven to fire fighters outside the building, etc. By defining a rigidpriority, the current Messages of those supervising the efforts to fightthe fire, or those in harm's way, are rendered ahead of those performinga less critical role. With non-tactical communications, a User may begiven a great deal of flexibility to define their own priority scheme tomeet personal needs. A sales executive may for example define a priorityscheme listing consecutive Conversations with clients from the mostimportant to the least important. Or a User may prioritize consecutiveMessages among family and friends. Regardless of the scheme used, apriority list for consecutive Conversations is defined (step 1210) inthis process.

Referring to FIG. 12C, a flow diagram illustrating the construction of aqueue of Messages received from the various consecutive Conversations isillustrated. In the initial step, the availability of non-renderedindexed media payloads of Messages (i.e., media streams) is continuallydetected (step 1212) for each of the Conversations to be consecutivelyrendered. The priority hierarchy is applied to the available indexedmedia payload streams (step 1214). Based at least partially on thepriority hierarchy, and possible other parameters as mentioned below,the available indexed media payloads are arranged into a sequence order(step 1216). The indexed media payloads are then continuously renderedin the sequence order (step 1218). By continuously repeating theabove-described process, the Messages of multiple Conversations areconsecutively rendered.

In one embodiment, the sequencing order is based either partly or fullyon the priority hierarchy. In alternative embodiments, other parametersin addition to the hierarchy and availability may be considered as well.For example, the sequence order may be defined using one or moreparameters such as the switching costs associated with interrupting acurrently rendered stream of indexed media payloads with the indexedmedia payloads of a higher priority Conversation, the quality of theavailable streams of indexed media payloads, the relative time theindexed media payloads were received, a shuffling order, or from inputof a controller of the system.

Typically when conflicts between the Messages of different Conversationsoccur, the indexed media payloads first in sequence order are renderedwhile the rendering of other available indexed media payloads are pausedor delayed. When there no conflicts, indexed media payloads areimmediately rendered as they become available.

In yet another embodiment, the Messages of the consecutively renderedConversations may be optionally reviewed in a time-shifted mode. If theUser of the first communication device generated any Media associatedwith the consecutively rendered Conversations, that media is indexed andstored in the PIMB 30 of the Device as well as the PIMB(s) 85 of theServers 16 on the network. Thus when a Conversation is reviewed in thetime shifted mode, the User has the option of reviewing just theincoming Messages associated with the Conversation, or both the incomingMessages as well as the Media created by the first User associated withthe Conversation in time index order.

L.2 MCMS-S Operation

In the MCMS-S or simultaneous mode, a User of a Client 12 enabled Device13 may define a set of Conversations for simultaneous rendering. Oncethe set of Conversations is defined, the indexed media payload streamsassociated with the set of Conversations are simultaneously rendered onthe Device 13, regardless if they overlap or not. In alternativeembodiments, the User may optionally render the received indexed mediapayloads from the set of media streams separately. The indexed mediapayloads of the simultaneous Conversations may also be optionallyrendered in near real-time or in the time-shifted mode.

L.3 MCMS, MCMS-C and MCMS-S Examples

In FIGS. 13A through 13D, a series of diagrams illustrating theattributes of a Conversation and the operation of MCMS, MCMS-C andMSMS-S is illustrated.

In FIG. 13A, a time diagram illustrating the sequence of rendering theindexed media payloads of the Messages of a Conversation labeled “A”between a User “X” and two other users, designated “Y” and “Z”. In thisexample, Media is generated by user Y during the time intervalsdesignated by t1, t5, 56, t7 and t9. Media is generated by user Z duringthe time intervals designated t3, t6 and t9 through t10.

The rendering sequence at the Device 13 of User X is illustrated at thebottom of the figure. During intervals t1, t5, and t7, Media derivedonly from Y is rendered. During intervals t3 and t10, only Media derivedfrom Z is rendered. In intervals t6 and t9, Media from both Y and Z isrendered. In intervals t2, t4 and t8, nothing is being rendered becauseneither users Y or Z are generating Media during these periods. Itshould be noted that intervals t1 through t10 are not intended to meanfixed time intervals, but rather, just periods of time when Media isbeing generated.

The diagram of FIG. 13A is useful in illustrating the attributes ofConversations. When one user is generating Media (either Y or Z), thatMedia is received at the Device 13 of X and is available for rendering.When both users X and Y are generating Media, both Media streams arereceived at Device 13 of X and is available for Mixing. When neitheruser X or Y is generating Media, no Media is received at Device 13 of X.As noted above, User X has the option of Reviewing the Media generatedduring Conversation A in either the near real-time or time-shiftedmodes. A1so the User X has the option of Reviewing the Media in theMixed format as illustrated, or to separately Reviewing the Media from Yand Z in the time-shifted mode.

FIG. 13B illustrates the operation of MCMS. In this example, the User isParticipating in three Conversations, designated A, B and C. ForConversations A, B, and C, the User either generates or receivesMessages designated (A1, A2, A3, and A4), (B1, B2 and B3) and (C landC2) respectively. The timing and duration of each Message is indicatedby the starting point along the time-line. It is useful to note that inthis example, all of the Messages overlap in time to some degree, exceptMessage B2.

With the MCMS application, the User selects one Conversation as current.For the selected Conversation, the User may Review incoming Messages andgenerate Messages which are transmitted to the other Participants of theConversation. In this example, the User selects in order Conversation B,C and A as current respectively. The Message sequence is thereforeinitially B1, B2, and B3, followed by C1 and C2, and then finally A1through A4. Again, while a particular Conversation is selected ascurrent, the User may transition between the near real-time and thetime-shifted modes and back again. The final rendering as illustrated inthe diagram is not intended to correspond in timing of the receivedMessages as illustrated in the top portion of the diagram. Rather thelower portion of the diagram is intended to show only the renderingorder of the Messages, based on Conversation order selected by the User.

The example of FIG. 13B is thus useful in illustrating the attributes ofthe MCMS application. A User selects one Conversation as current. Theother Conversations are paused. The User may also transition the currentConversation among all the Conversations at any time.

Referring to FIG. 13C, a diagram illustrating the operation of MCMS-C isillustrated. In this example, the User is Participating in twoconsecutive Conversations, A and B. With Conversation A, three Messagesare received A1, A2, and A3. With Conversation B, three Messages arereceived B 1, B2 and B3. It is useful to note that with this example,Message B1 conflicts with Messages A1. A1so Conversation A has a higherpriority that Conversation B.

During the consecutive rendering of the two Conversations, the higherpriority Messages A1 and A2 are first rendered in near real-time, asrepresented by the vertical dashed lines in the figure. Since there is arelatively large time gap between Messages A2 and A3, this space isfilled by time shifting and rendering Messages B1 and B2. When A3arrives, it is rendered in near real-time, while Message B3 is renderedonly after the higher priority Message A3 is rendered. By time-shiftingrendering of the lower priority Messages between the higher PriorityMessages, consecutive multiple conversations can be managed. It shouldbe noted that in this simple example, priority is the only parameterused to determine the consecutive order for rendering. As noted above, anumber of other parameters may be used as well.

Referring to FIG. 13D, a diagram illustrating MCMS-S is illustrated. Inthis example, a User is engaged in three simultaneous Conversations, A,B and C. The Messages A1, A2 and A3, B1, B2 and B3, and C1 and C2 arereceived for each Conversation A, B and C are shown in the diagramrespectively. With the MCMS-S, the incoming Messages are rendered at arecipients Device 13 as they are received. The rendering sequence of theMessages of the three Conversations A, B and C, as illustrated in thelower portion of the diagram, are therefore the same as when theMessages were received. In this manner, multiple Conversations may besimultaneously rendered.

In the aforementioned examples, several variations of the MCMSapplication were described, including MSMS-C and MCMS-S. Regardless ofthe specific type of MCMS application used, they all share severalcommon characteristics. In each case, the Conversations are defined by athreaded sequence or organization of Messages. The Messages aresegmented from a stream of Media, with each Message given a sequenceidentifier and indexed by the time the Media was created. Depending onthe variation of the MCMS application, Messages can be rendered inaccordance with one or more rendering options. The rendering optionsinclude, in one form or another, the filtering, grouping overlappingand/or serialization of Messages, using anywhere from zero to aplurality of different attributes. In this manner, multipleConversations, each including a string of Messages, can be conducted ona single Client 12 enabled Device 13. Lastly, each of the variations ofMCMS may handle the receipt of interrupt Messages in the same way. Whenan interrupt Message is received, it typically takes precedent over andis rendered before other Messages belonging to other Conversations.

M. Client and Server Hardware

Referring to FIG. 14A, a block diagram 140 illustrating the hardware ofa Device 13 used for storing and executing the Client application 12 isshown. The hardware includes a CPU 142, main memory 144 and mass storage146. As is well known in the art, the Client application 12 is loadedand stored in main memoryl44 and mass storage 146 and executed by theCPU 142.

Referring to FIG. 14B, a block diagram 150 illustrating the hardware ofa Server 16 used for storing and executing the server application 78 isshown. The hardware includes a CPU 152, main memory 154, mass storage156, and the archive 89. As is well known in the art, the serverapplication 78 is loaded and stored in main memory 154 and mass storage156 and executed by the CPU 152. As noted above, indexed media payloadsof one or more Users are stored in the archive 89.

Although many of the components and processes are described above in thesingular for convenience, it will be appreciated by one of skill in theart that multiple components and repeated processes can also be used topractice the techniques of the system and method described herein.Further, while the invention has been particularly shown and describedwith reference to specific embodiments thereof, it will be understood bythose skilled in the art that changes in the form and details of thedisclosed embodiments may be made without departing from the spirit orscope of the invention. For example, embodiments of the invention may beemployed with a variety of components and should not be restricted tothe ones mentioned above. It is therefore intended that the invention beinterpreted to include all variations and equivalents that fall withinthe true spirit and scope of the invention.

What is claimed is:
 1. A communication application embedded in anon-transitory computer readable medium when executed on a communicationdevice, the communication application comprising: a receiver moduleconfigured to receive at the communication device streaming video mediafrom two or more video streaming devices, the streaming video mediastreamed over a network from the two or more video streaming devicesrespectively; a rendering module configured to selectively enable therendering of the received video media from the two or more videostreaming devices in either a simultaneous mode or a consecutive mode:when in the simultaneous mode, the rendering module is configured tosimultaneously render the received streaming video media from the two ormore video streaming devices when the streaming video media issimultaneously received over the network, the rendering module includinga mixer configured to mix the received streaming video media from thetwo or more video streaming devices so that the simultaneously receivedstreaming video media from the two or more video streaming devices issimultaneously rendered; when in the consecutive mode, the renderingmodule is further configured to: prioritize the received streaming videomedia from the two or more video streaming devices in a hierarchicalorder; when the streaming video media from the two or more videostreaming devices is simultaneously received, sequentially rending thevideo media from the two or more video streaming devices in thehierarchical order; and when the streaming video media from the two ormore video streaming devices is received non-simultaneously atnon-overlapping times, rending the video media from the two or morevideo streaming devices in a time-order in which received; a mediacreation module configured to selectively create streaming video mediaon the communication device; a transmitter module configured to transmitover the network the created streaming video media selectively to one ormore of two or more video streaming devices, wherein the createdstreaming video media is transmitted as the streaming video media iscreated; and a storage module configured to store the received andcreated streaming video media; wherein the transmitter module is furtherconfigured to: ascertain bandwidth conditions on the network; transmitthe created video media as encoded if the ascertained bandwidthconditions on the network permit; and intentionally degrade a quality ofthe created video media when the ascertained bandwidth conditions on thenetwork do not permit transmission of the created video media as encodedand re-transmit at least portions of the created video media as encodedas network bandwidth conditions permit so that a complete copy of thecreated video media as encoded is eventually transmitted; and whereinthe receiver module is further configured to: ascertain if a completecopy of the streaming video media from the two or more video streamingdevices was received; and request missing portions of the streamingvideo media from the two or more video streaming devices not received;and store in the storage module a complete copy of the streaming videomedia from the two or more video streaming devices when the missingportions are received; wherein, the rendering module is furtherconfigured to selectively render in a time-shifted mode, the receivedand created streaming video out of storage.